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ABSTRACT 

This document presents seven papers prepared by 
experts in the field of adult education on the issues related to 
developing and implementing quality indicators from the perspective 
of their program areas or organizational affiliations. "Indicators of 
Program Quality for Adult Basic Education** (Hary Williams) addresses 
the appropriateness of indicators of program quality that were 
developed by Pelavin Associates for adult basic education (ABE). 
"Indicators of Program Quality: An ESL Programming Perspective" 
(Inaam Hansoor) examines Pelavin* s sample outline of proposed areas 
in which indicators would be addressed in English as a second 
language and some sample data elements for the indicators. 
"Indicators of Program Quality: Adult Secondary Education Programs" 
(Mary Ann Corley) suggests a repositioning of some indicators, a more 
specific detailing of others, and the addition of one new indicator. 
"Quality Indicators of Adult Education Programs for Local Education 
Agencies" (Connie Eichhorn) suggests additional indicators and 
dimensions more applicable to local agency programs. "Community 
College Delivery of ABE: Measuring Effectiveness*' (Donna Lane) 
describes the community college system in Oregon and appropriate 
indicators. "A New Framework for Assessing Program Quality: Meeting 
the Challenge of the National Literacy Act of 1991" (Sondra Stein) 
suggests a shift in the proposed approach to ensure proper evaluation 
of programs offered by community-based providers. "Indicators of 
Program Quality for Volunteer Literacy Programs" (Kevin Smith) 
suggests a set of program quality standards specific to volunteer 
literacy programs. A list of the sample quality indicators used as a 
guide by the paper writers is appended. (YLB) 



PROGRAM AND PROVTOER PERSPECTIVES ON 

DEVELOPING INDICATORS OF PROGRAl^ 
QUALITY FOR ADULT EDUCATION PROGRAMS 



Maiy Ann Corley 
Donna Lane 
Kevin Smith 
Maiy Williams 



Connie Eichhorn 
Inaam Mansoor 
Sondra Stein 



Prepared fon 

Office of Vocational and Adult Education 
U.S. Department of Education 



Pelavin Associates, Inc 
2030 M Street, N.W^ Suite 800 
Washington, D.C 20036 
(202) 785-3308 



July 1992 



u t ot^AnrtMtHT or coucation 



eouc 



' M.oof change) 



mem (Jo «nf 



BEST COPY AVAHASLE 



CONTENTS 



. . u 

Introduction 

Indicators of Program QuaUty for Adult Basic Education 

by Maiy Willianis 

Indicators of Program QuaUty: An KL Programming Perspective ^ 
by Inaam Mansoor 

Indicators of Program QuaUty: Adult Secondary Education Programs ^ 
by Mary Ann Corky 

QuaUty Indicators of Adult Education Programs for Local Education Agencies 

by Coimie Eichhom 

CommuniryCoUegeDeUveryof ABE: Measuring Effectiveness ^ 
by Donna Lane 

A New Framework for Assessing Program QuaUty: Meeting tiie 

ChaUenge of the National Literacy Act of 1991 ^ 
by Sondra Stein 

Indicators of Program QuaUty for Volunteer Literacy Programs ^ 
by Kevin Smith 

Resource Documents 

114 

Aopendix: Sample QuaUty Indicators 



ERIC 



DJTRODUCnON 



The National Literacy Act of 1991 requires the sutes to develop indicators of program 
quality for adult education programs by July 1993. To assist sutes with this process, the Act also 
requires the Department of Education to develop by July 1992 model indicators of program 
quality as guidance to sutes (Section 361(c) of the Aduh Education Act). The Act requires the 
indicator to be deveicpe^- lluough coosulution with eaqierts, educators and administrators of 
adult education. To develop the model indicators, the Department established a comprehensive 
process that included reviewing current sute indicators and evaluation criteria; reviewing 
indicators used by other Federal education and training programs; holding focus groups of a broad 
segment of aduh education providers, researchers, studenu and administrators; and consulting 
with sute directors of adult education^ Pelavin Associates, Inc. assisted the Department in this 
process. 

To assist in the indicator development process, the Department asked seven experts in the 
field - researchers, administrators and practitioners - to write brief papers on the issues related 
to developing and implementing quality indicators from the perspective of their program area or 
organizational affiliation. The experts represented the three program areas funded under Uie 
Aduh Education Act - aduh basic education (ABE), adult secondary education (ASE) and 
English-as-a-seoond language (ESL) and ti»e four main providen of instructional services Oocal 
educational agencies, community>based organizations, community colleges, and volunteer 
organizations). Pelavin Associates provided background materials and a preliminary list of 

*The model indicator development process and the indicaton are described in Model Indicators 
ofProgmm Quality for Adult Education Programs, avaUable from the Division of Aduh Education and 
Literacy Clearinghouse, U. S. Department of Education, 400 Maryland Avenue, S.W., Washington, 
DC 20202-7240, (202) 205-9996. 
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indicatOB to the paper writers to assUt them (see Appendix). The writers were asked to respond 

to the following questions: 

• Are there any unique aspects of your prtjgram area that should be taken into 
account in the development on indicators? What adjustments and adaptauons to a 
generic set of indicators may be necessary? 

• Are these indicators appropriate for your program area? Are any unnecessary or 
shoukl more be added? If so, specify which and the reasons the measure should 
be added or deleted. 

• What problems or other issues would confront your area when using these or 
similar indicators? 

This document presentt the seven papers prepared for this project The papen were first 
used by the Department to inform decisions about the model indicators and by participants in the 
focus groups as part of their preparations for the discussion. The papers will now assist states as 
they work toward developing their own indicatois. The Department gratefully acknowledges the 
invaluable contribution of these professionals. 
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INDICATORS OF PROGRAM QUALITY FOR ADULT BASIC EDUCATION 



Maiy Grdch Wniiams 
Mary Williams Associates 

The National Literacy Act. which amends the Adult Education Act, requires that by July 
of 1993. statss develop indicators of program quality for use in cvaluaung adult education 
programs. The indicators must be used to determme whether programs are effective. Three 
areas are cxpUcitly cited as areas of concern in the development of indicators. They are: 

• Recruitment; 

• Retention; and 

• Improvement m literacy skills of participants. 

The National Utetacy Act also charges the Secretary of Education with the rcsponsibUity 
of developing indicators of program quaUty that can be used as models by states and local 
programs. The indicators must take into account different conditions under which programs 
operate and be modified as better means of assessing qualiQr are identified. 

Pelavin Associates has defined an evaluation framework for adult education that is both 
comprehensive and flexible. The framework, published in July of 1991, is buUt around the factors 
that, according to federal regulations, must be "considered" by states in evaluating recipients of 
aduh education funds. The framework document presents multiple evaluation options on three 
levels of complexity and sophistication. 

In the introductory pages of the framework document, the assumption is made that three 
program components - context, processes and students - interact to produce measurable 
outcomes, and that an evaluation should (among other things) "Develop key indicators of the 
components of adult education programs, with emphasis on program outcomes." Subsumed under 
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headings that conespond to the three compoocnu are the factors that the regulations require to 
be considet Jd in evaluating recipients, along with other evaluative elements that pop up 
elsewhere in the regulations. e.g., use of standardized test data. 

Several reviewers, including myself, vxtt asked to review a document entitled "Sample 
Indicators of Program QuaUty." which extracts the major evaluation topics from the framework, 
expands or changes them somewhat (in part reflecting changes in the list of evaluation factors in 
the proposed new reguUtions), and sutes them in terms that arc measurable or observable. My 
response was supposed to address appropriateness of these indicators for adult basic education, 
problems or issues that wouU confront adult basic education programs in using them, and unique 
aspecu of adult basic education that should be taken into account including what adjustmenu and 
adaptations to a genetic set of indicators may be necessary. 

In attempting to make sense of the sample indicators or derivations thereof as models for 
consideration by the states, I found myself confronting various questions and evohong a set of 
principles to apply in selecting or developing indicators. This paper wiU address the assigned 
topics, whfle incorporating a strategy for thinking through this complex problem. 

The sample indicators appear to represent the "key indicators of the components" caUcd 
for in the framework. But is a "key indicator of a component" and a "quality indicator" one in the 
same? The material appeals to use them interchangeably. Semantics suggests to me that the 
former term wouU be wieftU in characterising or operationally deOning a program feature or 
requirement, or for identifying any dau that helps monitor, verify, describe, or ascertain the 
extent of a feature or requirement The latter term, however, suggests the dimension of value, 
merit or worth. For example, one of the items listed as a sample quality indicator is "existence of 
an advisoiy board." We can observe that an advisory board cxistt (indicator of the component 
•community input in program development"), but are we wflling to say that the mere existence of 
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an advisoiy board cootributec to profrain quality? FurtbenKMC are we willing to accept tbe 
-community input' component as a stand-alone m a nif rs t a tion of program quality? 

Hie evaluation Uteratuie makes a useful dtstiroion between a "primary indicator" and a 
•secondary indicator." In fact, Scxiven* sugfestt using tbe term "criterion" to refer to tbe former 
and "indicator" to tbe latter, to reduce oonfiisioa Tbe criterion, be says, essentially defines or 
embodies success or trutb, wbile tbe indicator is empirically connected to (a correlate of) tbe 
criteriotL In the previous example, "coaununi^ input" is tbe criterion (success/truth) «iiile 
"advisory board" is tbe (secondary) indicator if you can observe (empirically determine) tbe 
existence of tbe advisory board, tben tbe pcogram is successful on tbe quality dimension of 
"community inpuL" 

Ihe term "quality indicator" suggesu that we are talking about correlates of «^tever we 
agree constitutes success or trutb in adult education programs. There is considerable risk not only 
that <jve may come up with correlates that are of questicmable validity (perhaps because we do not 
have empirical evidence that they are consistently associated with tbe criterionX but that the 
"components" themselves may be questionable as what we want (in Scriven's words) "to count as 
thepayoCL" 

It k tempting to produce a laundry list of indicaton Cor each and every d im e n si on 
(component) of a program that might cnsL In fact, such a laundry list might have value as a 
guide to monitoring. Le., to establish a consistent way that sute and foderal program requirements 
can be met Or. it might serve a useful purpose as a program panning checklist, especially if it 
incorporates what k known about best practk« in tbe EiekL But when we get to the point of 
making quality (value) judgmentt about a program, judgmentt that not only may but perhaps 



'Scriven, Michael, Evalntkm ncsanras. Fourth Edition, Sage PubUcatwns, Newbury Paric, 
California, 1991. 
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$houW ifidrt the of students tad the lh«lihoods 

ultimate consequence. Questkins which cm gukle our thinking on this issue are. "b tto 
component the end we arc lookmg for or is it a means to get at something else?,* and "Does this 
component stand alone as a reflection of successr For eampte, is our payoff community input, 
or is it something more, suclx as whether the needs of afl target popuUtiom represe^ 
advisors are bemg addressed? Is ft sufiBdent Cor these to be addressed, or do they have to be 
met? Unless we define the outcomes we are looldng for in fonning an advisory board, how wiU 
we appropriate membershqt selections, define duties, and so on? 

Even if evciy ftem on our laundry list <rf components can be justified in some way, this 
approach tends to give every clement equal weight How many fKtots can a manager reasonably 
plan for, assess, fund, train for, bold dialogue on. implement, coordinate, collect dau on, and 
describe on a report form? How are priorities to be set? More importantly, how many of these 
activitiee are directly serve our main mission or purpote, as opposed to how many are means to a 
more important end? If we "require" multiple means as weU as ends, how sure are we that they 
are all necessary auf sufficient to reach the ends? Do they mduKtflWy lead to the end, *of«iim« 
lead to the end. or posstU)' lead to the end given a host of other circumstances? For cample, if 
a program uses ostensibly ettcDcnt recruitment strategies (diveise, targeted to special populations, 
makes use of media, interagency linkages, etc) but is not successful in actually recruiting the 
students, is ft an exoeUent program? 

If we can agree on. first, what in and of itself constitutes success (our criteria); second, 
how we win know those fkaors when we see them (indicatois); and third, bow much n^ 

there (standards), we will have the basis Cor a good evaluation process. 

I argue that, with only a few possible eweptions, outcome measures shouW be considered 
asourcriteria. Indicatois, then, oughi to be those factoit which best and most ck»cly meas 
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teprecent the outcooo. The other KJii-outcoine progrim compooentt (context 
presumabfy thoie that leseaich andAir pr«ak« ten w are 

But nctw $uicoiOtt^ about the degree and consistency with which these factois contribute to 
the outcomes to dedaie programs good. bad. effective, or ineffective on basis of their existence, 
separately firom the outcomes? 

ff we keep our criteria and indicators in the outcome realm, we can then look for 
existence of the instrumental or contributory factors as dues to understand differential 
performance OQ the outcome measures. If we amass enough data, we may find that there arc a 
few of these conttibutoiy facton that so reliably lead to the desired outcomes that we are willing 
to judge programs on the basis of those factors. More usefully, we can over time refine our 
knowledge about what ieatk to success, refine our planning tools, provide training and technical 
fTff«*«tif»^ that is targeted to making the ii|^t interventions, and fund processes and systems that 
are ^^^'^ to positive outcomes - while continuing to make judgments about program qualiQr on 
outcome-related performance. 

When we thinW of how 'conventional wisdom' has changed in adult basic education, it is 
dear that there can be danger in making judgmentt about program quality without defining the 
desired outcomes and researching whether interventions lead to the outcomes in question. Not 
too kmg ago, highly indivklualized and independent instruction was thought to be ind« 
best practice. Now, as research has shown the importance of the ability to work in 
interdependent groups, especially in the woriq>lace, programs are shifting to small group and 
cooperative learning. 

The hard part k, how do we aero in on the important outcomes? The obvious place to 
start B with the purposes that undergird the program and with aiiy cxplidtly stated program 
outcomes. 

5 
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The puipace of the Adult Education Act (paraphrased) k to improve educational 
opportunities fior adults who ladc the literaqr sldUs requinte to dtixettship and productive 
employment, and to improve services to educationally disadvantaged adults, through ptoframs that 
win. first, enable those aduhs to acquire basic sUBs for Uterate fimaianing; second, provide them 
with suffident basic shBs to ben^ from job training and to retain empkyment; and third, enable 
aduhs who so desire to complete the secondary school levtL 

Hk three areas cited in the National Ltteraq' Act as minimal considerations in the 
development of indicators of program quality (recruitment, retention and teaming gains) reflect 
measuraUe program resuhs. Indicators will need to he established for all three of these factors 
because of the requirements of the law. but because oofy learning gains could be d efen d e d as a 
meaningful end of instruction, making judgments on the basts of the other two fKton will be 
problematic Example: A program recruits 100 learners. Seven^ of them test below the sixth 
grade level, 20 test between 6th and 9th, and 10 above 9th. Let's assume that our standards 
reward this program in recruitment because of the large percentage of educa t ional ly 
disadvanuged and adult basic education clients it reaches. Let's also assume tLat our standards 
reward programs that retain students for at least 100 hours of instnictioa But our 10 hi|^ level 
students could pass the GED after 20-30 hours of stutfy. Do we give precedence to achieving the 
outcome of secondary completion, or persuade studentt to stay 70 hours beyond what is necessary 
in order to avoid bringing the retention rate down? Due care must be exercised i20t only in 
esublishing sensibk^ ladicatort and standards for such criteria, but in determining sensible rewards, 
sanctions and recommendations for practice based on measured performance. 

There are a few areas in which our criteria of quality may not meet the test of the 
•ultimate desired end.* For example, if we si^ that a program's enroUment should reflect the 
ethnic and racial diversity of the community, we would be justified on both moral and legal 



grounds. Meeting this type of criterion wouW be necessary, but not suffident for quality, the 
program would also have to meet other outcome criteria: We would want to know that our 
diverse population was not oidy recruited, but made learning gains and achieved goals. I want to 
point out that this criterion is nonetheless couched in terms of results (persons actually enrolled), 
and not of means (flyers distributed in ten languages). 

One other issue needs to be considered in this paper's attempt to develop a set of 
principles to use in the establishment of quality criteria and indicators. A distinction needs to be 
made between criteria iiWch are dearly applicable to all programs and clients, and those that are 
applicable to a subset of programs and/or clients. Of the nine items listed in the program 
outcomes section of the "Sample Indicators of Program Quality,' only two seem appropriate 
across the board: 'competencies attained" and 'achieved personal goals.' 

"Participation rates for population subgroups* will be applicable for most if not all 
comprehensive programs. But where multiple providers are intentionally funded so that each can 
address itself to a particular subpopulation, the criterion should not be applied. 

"Hours of instruction received" has aheady been esublished as an indicator that is 
contributoiy to other ends and as one for vAach standards may appropriately differ for different 
levels of instruction or type of instructional goal 

"Standardized test score gains" are generally deemed neither useful nor appropriate for 
complete nonreaders. In addition, progress tesU for certain subpopulations (e.gn ESL) are 
considered inadequate at best rmally, learners at the secondary level m^ be better served by 
tesU of mastery or certification (eg., GED) than by measures of skill gains. 

The set of employment-related outcomes are clearly reflective of the purposes of the 
Adult Education Act - but not all participants have employment-related needs and goals. The 
same could be said for secondary completion (not listed in the sample outcomes). 

7 
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The definition of model quality indicators is important in assisting at least some of the 
sutes to meet the tenns of the new federal legislation. Tbc models need to be designed 
intelligently and sensitively if they arc to be used and if ttey are to advance the field Hus paper 
has identified and grappled with a number of issues of importance in this process. TTiere is plenty 
of room for disagreement and debate, especially concerning the selection of parUcular criteria and 
indicators. A set of principles for selecting and developing quaUty indicators may aid the procas. 
One such set has cvoWed &om the exercise of preparing this paper. To summarize: 

• DififcrenUate between "primary" and Isecondary indicators. Consider using the 
term "criterion" to refer to the primary indicator, or the item we want to "count as 
the payoff," and quality indicator to refer to its observable manifesution; 

• Avoid creating a laundry list" of indicators for every possible program component 
or feature, or clearly separate "guidance" about means from "accounubility" for 
outcomes. Use the purposes and social values reflected in the law as a starting 
point for determining v^iul is most important; 

• With only carefully considered and justified eareeptions, keep criteria in the reahn 
of outcomes. Even where exceptions are made, focus on resultt rather than 
means. Choose indicators that closely and reliably represent those outcomes; 

• Design information management ^tems that keep track of contributory factors 
that appear to be supported in research and practice, and study their relationship 
to desired outcomes over time. Some may eventually be worthy of elevating to the 
status of criteria; and 

• Develop mechanisms in the evaluation process for giving more weight to factors of 
greater importance. Allow for the grouping of factors that may be compensatory, 
ic "any one these may be present to get credit," or "if this one isn't present, this 
other one is an accepuble substitute." Use differential criteria or indicators in 
cases where variable outcomes are allowed or encouraged. 
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INDICATORS OF PROGRAM QUALTTf: 
AN ESL PROGRAMMING PERSPECTIVE 



Inaam Mansoor 
Arlington Public Schools 



This paper was commissioned by Pelavin Associates Inc. to assist them in developing 
indicators of program quaUty for state aduU education programs. Pelavin is preparing these 
indicators for the U.S. Department of Education, OfBcc of Vocational and AduU Education. 
Recent amendments to the AduU Education Act require that indicators be developed at the 
federal level by July 1992 and by each sute by 1993. This paper examines Pelavin's sample 
outline of proposed areas in which indicators wiU be addressed and some sample data elements 
for the indicators. This paper is written from an ESL programming perspective. 

Pelavin has developed Sample Quality Indicators in the following areas: 

SECTION I: PROGRAM CONTEXT 

SECTION n. PROGRAM PROCESS AND CONTENT 

Part L Program Planning 

Part EL Program Process & Content 

Part m. Curriculum and Materials 

Part IV. Staff Qualifications 

SECTION m. PROGRAM OUTCOMES 

The following questions posed by Pelavin Associates are addressed at the end of this 
document (see Questions for CoBsideratloB) for each program area in which quality indicators 

are recommended: 

1) An then unique aspects of ESL imtram thai should be uUcm into 
devdopmenttfindieatonf What a4iustments and adaptations to a leneric set qf 
indicators nu^ be necessary? 

2) An these indicators apprapriaujor ESL programs? An arty unnecessary? ShouU arty 
be added? Which ones and why? 
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3) mat pnbum or other issues wouJdeotifmiaESLpntnM 
similar indieaton? 

INTRODUCTION 

In general, the development and use of quality indicators will be welcomed by the ESL 
field, particularly in Ught of the new federal guidelines which will further open up the delivery of 
ABE and ESL services to other providers. However, how indicators are formulated at the state 
level, what eicpecutions they cany, and what support wUl be made available for implementation 
wiU be the determining factors in whether indicators will be accepted as the means of improving 
program quality or viewed as another bureaucratic burden on aheady overstretched program staff. 
If the use of quality indicators is to be successful we need to consider the following: 

• Due to the wide diversity among ESL programs in terms of type of program, ^ 
program setting, learner goals and funding sources, it is impossible to have a "one 
size fits all' approach to quality indicators. Research has found that even among 
the best programs, there is no "one definitwc ESL or Literacy program" whose 
practices and procedures should be adopted by all.^ 

• Most programs have been operating on a shoestring. It will be diCQcult to 
compare programs and hold them accountable to an external set of outcome 
standards when some programs are not used to having outcome indicators of their 
own. Adequate funding to implement quality standards must be forthcoming. A 
sufficient period of time to raise programs to quality leveb will be required along 
with technical assistance and even esctemal evaluators. 

• The primary purpose of quality indicators should be to help programs define 
themsehres as educationally sound systems for the delivery of ESL and literacy 
instruction. 

Sections I & II of the quality indicators proposed by Pelavin can help with this definition 
of process. Section III seeks to use quality indicators to measure outcomes. 
SECTION L PROGRAM CONTEXT 

Indicators of quality in the area of "program context" would require that ESL programs be 
able to define the context in which services wiU be offered. This entails an assessment of need 
for the services, a descripUon of organizational structure of the delivery system and a definition of 
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the characteristics of the participants. Pelavin suggesu defining contexts through information 
gathered in the following areas: 

1. Need For Prognmi Services 

A. Number and demographics of target population 

B. Literacy Levels in community 

C High School drop-outs in community 

D. Employment related sidll needs of community 

2. OrganizatioB and Stnictare of Deliver? System 

A. Number of projecte 

B. Locations and settings of projects 
C Type of projects (e.g., ESUGED) 

3. Characteristics of Participants by sldU level 

A. Number and demographics of participants by skill level 

B. Number and demographics of participants by program type 
C Number and demographics of participants by program setting 

RATIONALE 

The process of defining context should begin with the development of a program mission 
and ideally a statement of the ESL program's driving philosophy. The mission statement should 
clearly and concisely define the program's purpose for existence. Yet, the statement must be 
bread enough to enable continued program growth and evolution through flexible and responsive 
programming. Once the ESL program undetsunds and articulates its own broad mission, it can 
make more rational decisions on program direction and use of resources. Furthermore, the sute 
agency can more objectively determine an applicant agency's appropriateness for delivering the 
proposed services, and determine among other things: 

• If the applicant agency understands the funding agency's mission and priorities 
(w^ch themseWes must first be dearly articulated to all); 



If the applicant agency has adequately defined its target population; 
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• If the Urget population b eligible for services; 

11 

17 



• To what extent the need for services exists; 

• How much of the need the applicant agency can realistically address; 

• If proposed services match the needs of the target population; and 

• The extent to which proposed services address a wide range of community needs 
and learner goals. 

Moreover* requiring ESL programs to provide information in these areas will strengthen 
the program's ability to make an assessment of the overall need for program services, and ii will 
help them to determine their "strategic market position in relation to other providers."^ This is 
even more important now as new providers enter the delivery system- Information required by 
the proposed indicators will help both new and existing programs to clearly deOne Vhat business* 
they are in and what services or benefits they can offer to specific constituencies. It will require 
programs to closely examme and reexamine their immediate program and environmental context 
By doing so, they will become better equipped to determine not only who they arc serving, but 
more importantly, who they are not sersring and whether or not thqr should be. This information, 
of course, will lead to more creative and dynamic programming because it will require regular 
assessment of needs and an evaluation of service response to those needs. 

The most imporunt aspect of ESL programs to consider is that they have ever changing 
populations. Programs should conduct periodic environmental scans to determine such 
information as: 

• Population trends 

• Economic development trends 

• Workplace trends 

• Legislative trends^ 

There are several other questions that ESL programs might want to investigate, such as 
what are the educational motivatoxs among the target population, how does the Urget population 

12 
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spend its time, when are participants available for instruction, etc Qearly answers to these and 
other questions will provide important infonn^ition to both the ESL program planners and those 
responsible for determining whether proposed services arc warranted. Realistically, however, 
because of time and budget constraints, programs should onty be required to demonstrate their 
understanding of the target population, the current institutional response from other providers in 
the service delivery area, and the extent of need for services by providing information such as the 
proposed Pelavin examples. 

SECTION n: PROGRAM PROCESS AND CONTENT 

Additional areas in which indicators of program quality are being considered are program 
process and content These areas relate to: program planning, program processes and content, 
staff qualificatioas, and curriculum and materials* 

PARTL Progran PlaBBlng 

Pelavin suggests that program planning should consist of such elements as: 

1. Community input in program development: 

A. Existence of an advisory board 

B. Program holds public hearings 
C Use of a needs assessment 

D. Other sources consulted (e.g., employers, staff, etc.) 

2. Coordination activities 

A. Existence of coordination arrangements 

B. Type of coordinated activities 

3. Written operational plan 

A. Existence of plan 

B. Specific program goals and objectives consistent with state plan 
C Plan development process 
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RATIONALE 

Program planning is a process which includes identifying needs, designing program 
activities, implementing those activities, evaluating results, and making further program decisions. 
This planning should resuit in a product ~ a written operational plan which enables all personnel 
to understand, analyze, and critique the program goals, objectives, and the strategies used to 
achieve them. It sets the framework for an educationally sound program. Ongoing analysis, 
' critique, and revision of plans enable programs to remain dynamic and responsive to new needs 
and population trends. It is tiiercfore a reasonable requirement for programs to not only be able 
to describe tiieir process for successful implementation, but also tiieir means of measuring that 
success wiUiin tiiose implemenution steps, Esublishing indicators of program quality in the area 
of program planning will lead to more effective implementation and informative evaluation 
systems for both programs and funding agencies. Requiring evidence of planning wiU also enable 
both the program and the state agencies to determine if a program has struck a very necessary 
balance between what needs to be maintained and what needs to be changed. Altiiough many 
programs have been in operation for a long time, it is imporunt thzt ongoing recvaluation of 
program needs and activities be conducted in order to determine what activities still meet the 
target population's needs and which ones have just become ritual In this way, program activities 
can be more targeted to current training needs, and resources can be used more effectively. 

What must be safeguarded, however, is a program's flexibility. An operational plan must 
not be so rigid or a program manager so concerned about being evaluated against the plan, that it 
cannot be amended in order to meet a more urgent or changing need. Program planners must 
remember Uiat Uiey are there for the learners, and policy at all levels must support Uiat 

Community input in ESL programs is also important There is a growing trend among 
ESL programs to seek out and involve representatives from the language communities that are to 
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be served, the learners themselves, and other sources of input (including major employers and 
potential employers) that have an interest in the target community. Some examples of 

collaboration and meaningful input are: 

• . Learner and/or teacher participation in governance, program planning & 

evaluation, curriculum, etc.; and 

• Collaborations with other literacy providers, workplace settings (unions, 
employers). 

Quality indicators should encourage these types of collaborations where appropriate and 
necessary. 

PARTn: PROGRAM PROCESS AND CONTENT 
Pelavin suggests quality indicators for the following activities: 

1. Recruitment; 

2. Program intake procedures; 

3. Ongoing assessment methods; 

4. Support services; and 

5. Exit and follow up procedures. 

RATIONALE: 

•n»e indicators of program quality that Pelavin suggests here are consistent with the work 
of other researchers who have identifled effective and educationaUy sound program processes. In 
the areas of program content, Rene Lerche concluded from her work on The National Adult 
Literacy Study (NALP) that effective adult Uteracy programs result from a systematic approach to 
program design and implemenution. "Successful programs have been designed as total education 
systems under which there is a balanced emphasis on (1) clearly suted learning objectives, (2) 
assessment of learner needs and progress, (3) instructional processes, (4) guidance «id counseling, 
and (5) program management and evaluation." Lerehe Onds that effective programs: 

• Are clear about their goals; 

• Have measurable objectives for each program activity; 
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Assist learners in determining if the program is suited to their goals; 
Have clear learner outcomes and standards for judging them; 
Diagnose each learner's needs and develop individual learning plans; 
Tie learning objectives to instructional methods, materials, and assessment; 
Provide feedback to learners on their progress and document progress; and 
Evaluate their program's effectiveness.* 
Aguirre International, in their US. Department of Education Study of Effective and 
Innovative Practices in Adult ESL Uteracy Programs, also examined eight program components 
similar to the NALP: (1) community outreach (2) needs assessment, (3) program design. 
(4) curriculum, (5) approaches and metiiods, (6) initial assessment and progress evaluation. (7) 
staff development, and (8) support services.' 

TTiese components are tiie nuts and bolts of an ESL program. They identify who is to be 
served, how they vwU be recruited, how their skills wiU be assessed for placement and progress, 
what support services wflU breakdown barriers to participation, and what wiU happen at program 
exit and foUow up. Indicators of program quaUty should be established in tiiese areas in such a 
way as to demonstrate to the funding agency that the program processes have been articulated 
based on the information that was forthcoming during the planning process and needs assessment 
The key to successful and responsive ESL program design is to strike a balance between what a 
program can offer (given its experience, resources and staff) and what the learners need and 
want QuaUty indicators should be structured to help shape this process. The following arc some 
issues to consUer within various program activities: 
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RccrnltmeBt 

Most ESL programs wfll have no trouble deOning their target populations, their 
recruitment methods, and their program outreach activities. Indicators of program quality should 
require that programs be able to demonstrate that they are targeting outreach efforts to learners 
for whom the program is appropriate and can show learners how the program can help them. 
Program Intake Procedures 

Program assessment procedures will, in some cases, have to be Ughtcned up and be more 
sharply defined to meet standards. Again, because of budget constraints, many programs have 
limited resources for testing and often students arc placed in class through an informal interview. 
At intake, procedures need to be set into place which allow for a balance between standardized 
tests (that allow for comparisons across programs) and program based alternative assessments that 
show where the learner fits into the particular program scheme. 

Individual learning plans (as a document) for low proficiency level students are not 
practical because the learners cannot articulate their goals with enough specificity to be 
meaningful. Conducting the interviews bilingually is also not possible for large programs with 
students from many different language backgrounds. While the process of helping learners 
articulate learning goals should begin at the lower levels (within the Umited language available to 
the student) individual learning plans should only be required of intermediate level and advanced 
students. 

Ongoing Assessment Methods 

Programs arc required by federal law to report standardized testing information as one 
measure of learner gains and program success. Unfortunately, that is the one measure which may 
be the least valid in determining those factors. To date, there is no one instrument which can be 
recommended for all program contexts. Nor is there any that is sensitive enough to record 
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general learner gains in the short periods of course time in which most ESL programs operate 
(40-100 hrs of non-intenswe instrucUon). ESL programs should be allowed to demonstrate that 
their assessment methods for determining student progress and learning gains are consistent with 
the type of program that they provide, and that their methods arc valid and reUable. Curriculum- 
based pre and post tests, competency based assessment or other alternative assessment systems 
which demonstrate gains in learning related to the instructional program and content will provide 
more meaningful dato than scores on standardized achievement tests. However, it is possible to 
use standardized test scores to establish and vaUdate program levels and determine program 
effectiveness among groups of learners as opposed to indhddual learner progress. The question 
remains which test to select and what is an appropriate sampling size and testing procedures. 
Technical assistance wiU surely be required. 
Snpport Services 

Programs should not be required to provide support services unless significant funds are 
available, ratiier they should be required to demonstrate their knowledge of where the services 
cxisj and how their participants are made aware of them. 
Exit and Follaw>Up Procedures 

Exit interviews or counseling sessions are reasonable and appropriate, but, of course, 
require a substantial amount of additional time and resources. 

ESL program participants are a highly transient population, and follow-up requirements 
(for each participant) to determine program impact would be difficult to meet However, 
programs should conduct surveys of program dropouts to determine wheUier the program services 
should be improved or changed. 
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PARTm. CURRICULUM AND MATERIALS 
Pelavin suggests indicators of program quality related to: 

1. Type of curriculum and instruction used; 

2. Materials and equipment used; and 

3. Selection and evaluation of materials and equipment 

RATIONALE 

Information about curriculum, materials and equipment used, and how those materials are 
selected is valuable, particularly at a time «*en the field is changing, new trends arc emerging, 
and the use of educational technologies is being encouraged. Aguirre International, has found 
that: "a strong curriculum is a conceptual framework that (1) outlines the kinds of Uteracy the 
program wants to achieve, (2) suggests approaches, methods, and materials; and (3) links 
claesroom teaching at the various levels with assessment and evaluation." * 

Since ESL learner needs and goals are broad and vary from program to program, the sutc 
should not necessarily mandate a standardized curriculum or mode of instruction, but guidelines in 
effective practices are appropriate. Some sutes have already provided direction to programs on 
methods and approaches, e.g., CaUfomis stresses lifeskills instruction and language acquisition 
over "language leamtog", other states are now identifying approaches that work best with second 
language learners (New York), and others have developed or are developing curriculum guides 
(Washington, Texas and Florida).' At a minimum, sutes should require that programs define the 
linkage bebween the needs assessment and the curriculum. Programs should esublish their 
curricula first in terms which are appropriate for their learner goals and then in terms of learner 
needs within those goals. States should also demand that the approach is educationally sound and 
reflects an understanding of what we know about second language acquisition, adult literacy and 
how adults learn. 
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Based on a review of the literature and on input from researchers and practitionen in the 
field, the Aguirre study showed that educationally sound ESL Uteracy lessons shared the foUowing 

characteristics. Th^are: 

Interactive (students talk to each other tod to the teacher, students are actively involved 
in generating writing and interpreting vAiui others have written); 
Responsive to multi-levels (students work togeUier as a group or in pairs; activities work 
for different student levels); 

Learner-centered (the type of language and literacy Uught supports the goals of the 
learners and builds on their personal strengths and their life experiences; students have 
the opportunity to make choices during the lesson; classroom activities are linked to 
learners' lives outside of class); 

Meaning-based and communicative (the activities that students are engaged in reflect 
language and literacy use ouuide of the classroom;there is a point to the lesson other than 
literacy practice"); and 

Integrated in respect to language skills (reading, writing and oral language use are 
connected; conversations and discussions lead to reading and writing and vice versa). 

Th^ also provide a balance between activities that focus on communications and the 
expression of ideas and those that emphasize language awareness.^ 

A strong ESL curriculum will also be organized by instructional levels with specific learner 
outcomes and related language skills. Additionally, a strong curriculum will define the methods 
and materials used for implementing the progranu Most importantly, the methods and materials 
used by ESL programs should reflect an understanding of how adults learn and should provide for 
differences in a learner's ability and learning styles. 
PART IV: STAFF QUALinCATIONS 

Pelavin proposes indicators of quality in this area as follows: 

1. Characteristics of staff 

Z Staff responsibilities 

3. Staff development 

4. Use of volunteer suff 

5. Evaluation of staff performance 
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RAnONALE 

Implementotion of a quality ESL program is dependent on the staff hired to conduct 
program activities. Yet, in the past, it was a commonly held view that "if you can speak English, 
you can teach English". It is difficult to say whether this view is the result of common 
employment conditions today or the cause of them. Currently, like the Odd of adult literacy in 
general, the majority of adult ESL programs in the country arc staffed by part time teachers, some 
of whom arc trained professionals, others are trained in other Oelds and others arc volunteers.' 
There is an ongoing debate regarding Vho is an adult educator?" and "who educates adults?" 
TTiis debate cries out for the professionalization of the Odd of adult education and the teaching 
of English to adults. The indicators which Pelavin has suggested are appropriate and further our 
goal of professionalization. It would be hdpful to sec minimum stafOng standards rdated to 
training, full/part time sutus. staff evaluation, and staff input into program design and evaluation. 
These minimum sundards must be backed by the resources necessary to implement them. 

A discussion of staff qualifications, however, must recognize the fact that there is no 
agreement in the field over appropriate qualifications for ESL literacy teachers. Minimum staffing 
standards may really work against a number of community-based organizations. Despite this 
conflict, if wc arc to improve quaUty of instruction, guidelines need to be estoblished to assist 
CBO's in improving thdr staffing situations as well TESOL's Standards and Self-Study document 
should be consulted in the development of standards for this area." 

Given this ongoing debate, standards for staff development wfll need to take a more 
prominent role for ESL programs. Staff development should be based on the needs of the 
instructors, the needs of the learners and the needs of the program. Standards wiU also need to 
be fladblc enough to allow for changes that arc occurring in staff development such as more 
teacher centered approaches, where the teacher is hdpcd to meet his/her own goals. This 
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individutUation of staff development is a very powerful means of meeting the needs of teachers. 

learaecs and programs. 

Staff responsibUities is another issue that needs to be closely considered. Teachers should 
not be expected to volunteer their time beyond the paid classroom duties. The quality indicators 
that are being proposed vwll impact on them by requiring additional responsibiUties for counseling, 
data coUection and program input They should be paid for thcr- additional responsibilities. 
Most importantly the should be paid for planning time. Paying for planning time in itself sends a 
message to the practitioner that their endeavor is a profe»ional one requiring an assessment of 
their individual student needs, an adaptation of the program curriculum to meet the needs and an 
integration of appropriate instructional and evaluation techniques and materials to implement the 
curriculum effectively. 
SECTION m: PROGRAM OUTCOMES 

Pelavin has suggested indicators for the following program outcomes: 

1. Retention 

2. Educational gains 

3. Employment 

4. Goal achievement 

RATIONALE 

These indicators reflect the need to determine both impact on ESL learners and provide 
data for program accountability. No one would dispute the fact that accountability is necessary 
and justified If we are designing and implementing sound learning systems Uien we should tec 
successful outcomes. The problem then remains to identify and use appropriate indicators and to 
separate out the purpose for which the data for the indicators wiU be used. Ruth Nickse in her 
report, "A Typology of Family and Intergcnerational Uteracy Programs: Implications for 
Evaluation", uses a framework that identifies five levels of evaluation that she applied to various 
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types of family literacy programs. The following chart is an adaptation of this framework (for the 
purpose of eaounining Pelavin's quality indicaton).^^ 



1 LEVELS 


LEVEL 1: 

NEEDS ASSESSMENT 


LEVEL 2: 

ACCOUNTABILmr 


LEVEL 3: 
PROCESS 
CLARinCATION 


LEVEL 4: 

PARTICIFANT 

PROGRESS 


LEVELS: 1 
PROGRAM IMPACT 1 


QUESTION 


b there a seed for the 
profnuB? 


What will the progfim 
<k> to neet the Med and 
how can it be nonitored? 


How can the pcognuD 
iaprcwc iu tervioei? 


Are panicipanu making 
piuficai? 


What are the kMig lenn 
eftecu o( profraB 
partidpanu? 


DATA 

ELEMENTS TO 
CONSIDER 


*dcnofnphic infoniMtion 

OOStflOS 


Hriittca operatioaal pUn 
•fiuCfiirT 4cotcat 
*citniculua aod 
autcriali 

•hipata: MfLoonauniiy, 
adviiocy boacdt 
evaluator, fuadini 
toune 


•participation ratet 
•letcatioo fates 
*piT)fraai ooapletion 
*dient aatkCactioQ 


•proCciency pint 
•levclgaina 
•goal atutnmeat 


*a»tinuia| education 
*iBprawid job or 
iaoooK 

*k»g tetu foaU met 



Most adult ESL programs should be able to meet at least the Orst four levels of evaluation 
by providing the suggested outcome data. TTie fifth level, long term program impact as 
represented by Pelavin's suggested "employment" related indicators may not be possible to collect 
accuratefy. When it is collected and reported for state reports, the information may not be 
correct because it is often second-hand information from other students. Employment related 
outcomes, while interesting, cannot truly be attribuuble to general ESL program, even if valid 
data collection in this area were feasible. 

The employment related outcomes that Pelavin suggests are related to what Hal Beder 
described in his new book. Adult Uteracv Issues For Policy and Practice as the "human capital 
theory of program impact" The human capital theory of impact provides justification for federal 
expenditures on adult literacy instruction, the theory being that improving literacy and basic skills 
will lead to "improved worker productivity which in turn leads to improved national productivity 
and increased national wealth." Beder lists four components to human capital impact: increased 
employment and increased quality of employment, increased income, reduced need for public 
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asstttance. continued investment by the student ia further education. Bcder noted several studies 
which showed human capital gains, but notes that without experimental control groups, there is 
no way to know with certainty that those gains were a result of aduU literacy instruction. Beder 
states that when reasons why adults participate are considered, it is clear that most are striving to 
improve themscNes. Bcder recommends that "The federal adult Uteracy education program be 
held accountable for the broad personal and social development of its clients rather than to 
narrow human capital outcomes".^^ 

Many programs have valid goals that are not directly related to employment such as 
strengthening Uteracy practices at home (family literacy); helping learners to communicate in 
English, access services, and gain greater independence (life skills programs), promote greater 
participation in the democratic process (community literacy and civics classes)preparing to enter 
GED, college or vocational programs. 

Much of the literature rrmintains that increased self-esteem, stronger decisionmaking skills, 
and effective strategies for learning how to learn are important outcomes of language and literacy 
programs. Programs that include these domains in their framework should be encourage to show: 

(1 ) How their efforts link linguistic goals with non-linguistic goals; 

(2) What educational opportunities are provided to reach these goab; and 

(3) How staff plans to evaluate whether (and to what extent) these goab have been metP 
Therefore, given the fact that we cannot directly attribute human capiul gains to general 

ESL programs, given the need to also impact on the social and pereonal goals of learners 
(because that is their primary reason for participation), and given the limited resources that 
programs have, isn't it more important to measure outcomes over which we have control Le, 
retention, educational gains and goal achievement? These three areas are within a program's 
control over data, and outcomes can be directly attribuuble to program processes. 
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Making comparisons of outcomes across programs, however, wiU still be difficult since 
effective ESL programming is a dynamic process. Outcomes are based on individual program 
focus, learner goals, educationally sound program curricula and practices, quality of classroom 
instruction and evaluation procedures. No two programs are the same. Quality indicators should 
take this into account and allow for program-based assessment. 
Summary 

While the use of quality indicators is appropriate, we must be realistic in our expectations. 
After conducting research on over 200 programs nominated for their promising practices, Wrigley 
and Guth have found that it is not possible to identify "one definitive program to model". In their 
forthcoming manual. Adult ESL Literacy: Issues. Approach es, and Promising Practices, the 
researchers state: 

Given the large variety of programs and the diversity in program focus (general, 
workplace, family, community Uteracy) and program goals (self-sufficiency; acculturation; 
academic.) 'establishing standards" might require that we emphhsize processes that help 
ensure quality instead of focusing on outcomes or products. That is, programs should be 
held accountable for having in place structures, plans, and eva luation tools, that promote 
quality education. 

For accountability's sake, programs will be asked to show how they: 

• Plan to implement a quality program; 

• Define success" (both in program terms and related to learner outcomes); and 

• Will evaluate program success and judge learner progress plan to develop their 
own Standards and help ensure that those will be met 

Accountability will be based on the quality of: 

1) Their planning process; 

2) Their effort to provide quality service; 

3) Their flexibility and responsiveness to learner needs; 

4) Their evaluation efforts, geared toward both program improvement overall 
and assessment of learner progress and performance; 
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5) The actual changes made based on the evaluation results; and 

6) The establishment of quality indicators based on program experience over a 
reasonable time.^^ 

While this may not be a popular approach to take in defining program accountability, it 
may be a more realistic one (given the state of the art and the unique characteristics of ESL 
programs which on the one hand enables them to be responsive, and on the other hand impedes 
comparison of programs against each other). 

When finalizing quality indicators, it is essential that the following general characteristics 
of ESL programs be kept in mind: 

1) There is wide diversity among ESL programs in terms of service delivery 
settings* in terms of program focus, and in terms of funding sources. 
Qiiality indicators must be broad enough to capture the essence and the 
reality of these varied programs and flexible enough to enable them to be 
responsive to their client population and to the many other different 
requirements that they must be accountable for (funding sources, parent 
organization rules and regulations, eta); 

2) Quality indicators must be written in such a way that thqr promote 
responsive programming not inhibit it Programs must continue to be able 
to be responsive to the wide range and varied needs and goals of their 
learners; and 

3) ESL programs have limited resources. The spirit of the indicators should 
be such that it encourages and promotes quality services to the leamen. If 
accounubility requirements become so cumbersome, and monitoring so 
rigid, as to cause programs to spend an unreasonable amount of valuable 
lime and resources to meet those requiremenu, then quality will suffer. If 
programs are to meet and maintain a minimum set of standards, theri 
additional time, resources and technical auistance must be made available. 

The field is changing and quality indicators must allow for new trends to emerge and for 
new approaches to be tried Quality indicators must Uke into account the creative and 
experimental nature of programming for effective and innovative ESL and literaqr programs. 
Quality indicaton must enable the emergence and experimenutton with such trends as: 

• Participatory education: Programs in which learners have a role in program 
implementation, curriculum, and assessment; 

• Learner strategies training: An approach which encourages cognitive awareness 
and control over learning; 
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• Educational technologies: TTiere is still much to be learned about the potential 
that new technologies hold for language teaching; and 

♦ Functional context instruction: An approach to training which narrows learning to 
specific content areas 

TTie pluralistic and experimental nature of the field of aduh ESL must be safeguarded or we risk 
the loss of responsive education and the ability to grow and expand knowledge and experience in 
exploring innovative and effective ways to provide instruction. 
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QUESTIONS FOR CONSmERATION 



SECTION 1: Program Context 

1) Art there unique aspects of ESL programs that should be taken Into account In 
the development of Indicators? What adjustments and adaptations to a generic 
set of indicators may be neccssaiy? 

ESL programs are unique in having a greater need for assessing information which can 
help them defiiic their ever-changing and growing populations. This ever-constant need to 
evaluate context and make programming decisions based on the context defines the very "fluid 
nature of ESL programming." However, this characteristic of good ESL programs will make it 
difficult to evaluate programs against each other. The immediate implication is that programs 
should be evaluated against how they have defined tiieir contexts, what processes they have 
established to be responsive and what outcomes tiiey have established for themselves." This is 
surely not a poUtically popular implication, but it is the reality of the nature of ESL programs. 

Concerning information on "the number and demographics by skiU level" programs should 
have formal descriptions of proficiency levels and should define the relationships among their 
placement testt, instructional levels, and expected outcomes." Ideally, tiie levels would be 
esublished or correlated to nationally recognized systems. This can be done by either establishing 
common skiU level definitions at the state level or enabling programs to give their own definitions. 
There are several possible resources for defining levels; the Mainstream English Training 
Program's Studect Performance Level Descriptions," CASAS Scores" or BEST test 
scores." Several sutes already have skUl level descriptions defined by the previously mentioned 
systems. However, it might be more advanugeous and realistic for programs to be required to 
define their own skiU leveU and demonstrate that they arc appropriate for the type of learner and 
ESL program services that are being offered. Perhaps tiie first step is to establish consistency and 
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progression within a program before we try to compare programs. If programs define their own 
skfll leveb, those definitions should clearly indicate both the general language and specific 
language skills that are represented at program entry and/or are expected at exit 

Skill level descriptions will obviously differ between programs that focus on teaching 
English for Ufeskills communication and programs that focus on workplace literacy skills, or pre- 
academic skills. 

2) Alt these indloitors «pproprl«te for ESL programs? Are any unnecessaiy? 
Should any be added? IVhieh ones and why? 

The information requirements proposed are appropriate and should be considered 
baseline. Ideally, programs will fmd out more information than is being required here. A 
statement of program mission should be added. A broad mission statement will help the program 
determine how it percewes itself, how it wants to be percewcd and whom it wants to serve. In 
this way it waU help the state determine if proposed services are appropriate for the proposing 
organization. 

"Literacy levels of the community" should be changed to education levels of the 
community. 

The literacy levels of the community" is confusing. Who is the community? Those who 

are receiving services? The general population? The proposed target population? How would 

one gather data on the literacy levels of the community? Perhaps this should be changed to 

•education data on the community", which can easily be gathered through such means as census 

dau and economic devetopment data. Information on subpopulations may also be possible, such 

as the number of people in the census tract who report that they do not speak English well 

3) Problems or other Issues which would confkOBt ESL programs when aslag these 
or similar indicators? 
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dear definitions of terms within the indicators will be required. For example, does 
-community- refer to the community in general or the target population? Does -skill level" meaii 
individual language skills (reading, writing, listening, speaking) or does it mean general proficiency 
level? Perhaps skill levels woukJ even be defined more broadly to include knowledge of U.S. 
culture or content such as workplace skills, citizenship information, knowledge of the community, 
etc that a learner is expected to have or expects to develop in the course. 

Concerning -Characteristics of participants by skiU level" ( data requiremenu on 
participant), in order to facilitate information to programs seeking data information and to avoid 
dupUcation of effort, it will be necessary to have a reporting process which will facilitate easy 
access to information on learners and programs that are currently in the system. The state's 
annual report to the U.S. Department of Education can serve as one valuable source of 
information for local providers as weU as for the Department of EducaUon. It should also be 
noted that programs may not be famiUar with sources for data requirements and some guidance or 
technical assistance may be necessary. 

Last, but not least, is the fact that ESL programs arc generally poorly funded and 
primarily staffed vwth part-time personnel. Although many programs conduct the aforementioned 
activities, the information may not ahvays be in a format thni is meant for outside consumption. 
SECTION II: Prognrai Process and Content 
PARTI: Program PUwniBg 

1) Ai« there valque aspects of ESL programs that should be Uken Into account ia 
the dcvclopneat of ladicators? What a4|ustmeats and adapUUons to a geacric 
set of ladicators may be necessary? 

Although Pelavin's indicators are reasonable for this area, it can be expected that there 
may be many programs that would not be able iz produce written evidence of these activities 
because of time and resource constraints. Good programs, however, do conduct these kinds of 
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planning activities. It is the planning and evaluation process that is most important The task of 
developing an overall operational plan may seem overwhelming at first, but programs should be 
encouraged to organize and compUe materials produced from the planning process into a 
systematic operational plan and demonstrate how that operational plan was developed, how it will 
be evaluated, and how it will be revised. 

If programs establish an overall operational plan, then the next logical step is technical 
assistance for formative and sununative evaluations of the plan. 

2) Arc these iBdicaton ■ppropriatc for ESL prognBU? Are any uBBccessaiy?, 
Should any be added? Which ones and why? 

Pelavin suggesu a need for quality indicators for Program Planning related to community 
input in program development, coordination activities, and written operational plan. Each of 
these areas is appropriate in that they take into account the who, what, where, when and why of 
planning. All ESL programs should conduct planning and coordinating activities. However, 
"Holding Public Hearings' may be an indicator which is signiGcant only if the state or federal 
agency requires it Less formal focus group discussions with current and potential clientele 
usually are far more desirable and productive. Holding public hearings for ESL populations 
would also require bilingual assistance, and in most programs there are numerous language groups 
represented. 

3) What problems or other issues would confront ESL programs when using these 
or similar iudicaton? 

As ahvays, time and resources will be a problem for ESL programs. The activities 

described above are clearly administrative. Adequate resources and technical assistance must be 

made available if these activities are to be required. 
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PartD: Prognm Process and Content 

1) Aw there unique aspects of ESL prognuau that should be taken into account in 
the development of indicators? What adjustments and adaptations to a generic 
set of indicators may be necessaiy? 

There is wide diversity among programs offering ESL instruction. This diversity includes 

type of program, program focus, program approaches and phflosophies, funding sources, etc. 

There is also a wide diversity in learner goals and needs. This diversity from both programming 

and learner needs perspective must be uken into account There must be a balance between the 

need to control and regulate quality services and the need to aUow for creative, innovative and 

responsive programming. 

2) Are these indicators appropriate for ESL programs? Are any unnecessary?, 
Should any be added? Which ones and why? 

ADJUST 

The indicator that wfll be the most problematic for ESL programs and learners at the 
lowest proficiency leveb is "development of individual learning plans*. It is not the process that 
will be challenged, but the product. Many practitioners will tell you that at this level "an 
individual learning plan* is an empty exercise m paperwork. Most practitioners wiU agree that 
leamen at the lowest proficiency levels cannot express goals in English and indeed the concept is 
likely to be culturally alien to them. Forcing very beginning language learners (particularly 
Uteracy students) to set goals that may be too high and unrealistic can fiustrate them and cause 
them to drop out However, most practitioners recognize that it is important to help learners 
take charge of their learning by enabling them to begin to discover and articulate their goals. "A 
process for accomplishing this must keep two issues in mind, (1) learners new to speaking English 
may not be able to articulate their goals (especially at the beginning course), and (2) learner gpals 
may change as they gain greater confidence and increase their proficiency.*** For learners at 
intermediate and advanced leveb, individual learning plans may be more appropriate instrumentt 
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for goal definition than for the lower proficient students. For them, perhaps more manageable 
learning goals (Ic., life skills competencies) that the learner can achieve and experience success 
with will be more favorably received by both learners and instructors and will enable them to 
begin the process of goal articulation. 

3) V^t problems or other Issues would confront ESL programs when using these 
or similar indicators? 

Discussed in #1 above. 

Part ni: Cnrricnlnm and Materials 

1) Are there unique aspccto of ESL programs that should be talien into account in 
the development of indicators? What adjustments and adaptations to a generic 
act of indicators aaay be acccssaiy? 

ESL populations, unlike ABE populations are ever changing and programs must be able 
to recognize new population trends, assess their needs and respond with new and appropriate 
curricula that are linked to the changing needs. It is reasonable for a state agency to require 
programs to define their instructional program in such a way as to enable the state to determine 
whether content is based on learner goals and needs assessment as well as whether the processes 
are appropriate. For example, if a program determines that a large number of prospectwe 
learners need to learn English in order to function in daily life, a grammar-based syllabus would 
not show a relationship between learner needs and the proposed plan of instruction. Similarly, a 
program that has as its goal to help parents support the schooling of chUdren, needs a curriculum 
that includes strategies for understanding and interacting with the U.S. school system. 
Responsiveness to context is the key for determining appropriateness of curriculum and 
approaches. 

2) Ai« these indicators appropriate for ESL programs? Are any unnecessary? 
Should any be added? Which ones and why? 

The indicators are appropriate. 
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Most adult ESL programs still do not have accas to adequate equipment Cor instruction. 
As our communities and work places become more technologically advanced, it is increasingly 
important for adult learners to use educational technologies. Adult ESL programs should begin 
to at least plan for the use of educational tcchaologies and explore ways in which the equipment 
might be acquired. 
SUGGESTED ADDITION 

a) Methods and materials should be educationally sound and reflect an 
understanding of how adults learn 

b) Program has developed a plan for using and integrating educational 
technologies 

3) y/h»t problems or other issues would confiroBt ESL programs when using these 
or similar indisators? 

Tunc and resources will be required to truly achieve this standard. ESL teachers are very 
creative individuals and the field would benefit from additional allocation of resources to 
programs for classroom based research, teacher mput in programming and evaluation, and 
development and dissemination of materials. 
Part IV: Staff Qnallflcatioiis 

1) Are that nnique aspects of ESL programs that should be taken into account in 
the dcvetopment of indicators? What ai^ustmeBts and adaptations to a generic 
set of indicators may be necessaiy? 

All of the indicators specified by Pelavin are commendable and would indeed demonstrate 
quality in the area of staff qualifications. However, there is a unique aspect that should be token 
into account - the largely part time stoffing situation of most ESL programs. This severely limits 
then- availability for stoff development and for more staff input and invoWement in program 
design and evaluation. Additional funds for these acthrities must be forthcoming. 

2) Are these Indicators appropriate for ESL programs? Are any nnnecessary? 
Should any be added? Which ones aad why? 
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The indicators are appropriate with adequate resources to support them. Paid pre-service 
and inservice as well as pay for non-classroom program input are essentihl elements of quality 
indicators for staff development 
Sugsested Addition 

a) The term "Su£f qualifications" as opposed to "staff characteristics" should 
be used in the indicators. Tliere is a specialized body of knowledge about 
teaching English as a Second Language that has been developed and 
articulated. ESL teachers should have training in the study of language, 
second language acquisition theory, adult learning theory and ESL teaching 
methodology. 

b) Paid Planning time should be included as an indicator of program quality. 
Paying for planning time recognizes this as a professional activity and u 
consistent with practices in other educational settings such as the public 
schools. 

c) Use of volunteer staff should be viewed as a supplement to paid 
instructional staS. 

3) What problems or other issues would confront ESL programs when using these 
or similar indicators? 

Most adult ESL programs have chosen to allocate funds toward services at the expense of 
full time suHing. This is due to the burgeoning need for services that administrators face daily. 
Most ESL programs could not function without the hard work and contributions of a trained part- 
time workforce and a volunteer force. However, if programs are to be measured by new and 
stringent quality indicaton, it must be made clear to the funding agency thai professional 
standards require a sUble and professional base sUff. As Hal Beder states, "Qearly, if adult 
literacy education is worth doing, it is worth doing well. This requires a well trained, well paid 
professional workforce. To this end, reliance on part time teachers and volunteers is 
anathema."^' 
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Section m: Prognn Outconet 

1) Ar there unique aspects of ESL programs that should be taken Into account in 
the development of Indicators? What adlJustmenU and adaptations to a generic 
set of Indicators may be neccssaiy? 

Limited time and rcsourcct , open entiy/open exit procedures (which are typical of most 

ABE/ESL programs), and lack of good sundardized test measures, diversity of program types, 

settings, and goals are all aspects that should be taken into account when setting indicators. 

Wrigley and Guth suggest that: "Given the diversity of programs and the resources that have 

been expended on ESL, it may be unreasonable to expect programs to meet externally defined 

standards. It may fint be necessary to require programs to make all reasonable efforts to (1) 

improve their services (many may need guidance) and (2) provide documenution and what they 

have done and why, and then (3) develop their own standards given their particular realities."^ 

2) Are these Indicators appropriate for ESL programs? Are any unnecessary?, 
Should any be added? Which ones and why? 

Indicators in the area of retention, educational gains and learner goals are appropriate. 

Indicators in the area of employment changes are not appropriate for general adult ESL 

programs, because of reasons described in rationale section. An additional problem for ESL 

programs in inquiring about employment is the issue of confidentiality. Many programs are 

funded by non federal dollars and there are probably many undocumented aliens participating in 

the programs who would be reluctant to give employment data. It would be disruptive to program 

procedures to require employment information for some of the students and not from others, 

particularly when there is very little confidence in the information's accuracy and relevance. 

3) What problems or other Issues would confront ESL programs when using these 
or simlhu- Indicators? 

Standardized testing remains a problem for adult ESL programs. Although there is a 
federal requirement for standardized testing, it should be recognized that there are no 
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standardized tests which can be generally applicable to all types of ESL programs. Most are not 
seasitive enough to measure general language proficiency gains in short periods of time (especially 
for low proficiency levek), and most are very lengthy and costly to administer. Additionally, most 
standardized tests focus on individual facets of language, listening, speaking, reading, writing or 
grammar. This one-dimensional approach does not capture the holistic nature of second language 
learning. Thomas Sticht. in his report, Testing and Assessment in ABE and ESL Programs." 
reports that there is serious concern from the Geld about the federal requirement for standardized 
testing.^ He suggests, "GeneraUy, in testing in ESL programs, as in other ABE programs, it 
may be desirable to separate testing for program accountability from testing for instructional 
decision making."" This is a valuable suggestion and programs should be aUowed to submit 
other data on learner gains which will yield more meaningful data related to the outcomes of 
instruction. 

Learner gains in ESL should also be considered in Ught of various program and student 
related conditions. Research conducted by the MELT project identified program related factors 
such as intensity of instruction offered, program curricula, trained staff, etc., had an impact on 
amount of learner gains made. While these quality indicators wiU help address some of these 
program factors, there are other student related factors which impact on learning gains (age, 
previous education, previous language experience, physical abilities, etc.) 
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INDICATORS OF PROGRAM QUALITY: ADULT SECONDARY 

EDUCATION PROGRAMS 



Maiy Ann Corley 
Maryland State Department of Education 

QuaUty for Adult Secondary Education (ASE) programs can be defined by the presence of 
key functions (indicators) which. coUectively. best support adult learners in acquiring a secondary 
level education. For each indicator or key function, there must be evidence that minimum 
standards are being met before "quality" can be said to exist 

It is almost impossible to create a list of quality indicators (key functions) without defining 
or making reference to the standards against which the indicators or functions are to be 
measured The delineation of standards is critical not only to cvaluators for determining the 
degree to which quality cjdsts within a given program but also to directors and planners for the 
continuous monitoring and strengthening of their programs. Even among those programs which 
meet minimum standards, there arc varying degrees of "quality." with some programs 
demonstrating greater evidence of quaUty than others. Planners and directors who use the 
standards list as a guide for strengthening key program functions automatically engage in the 
process of improving overall program quality. Therefore, it is essential that t he standards list be 
detailed and specific. 

TTie sundards are developed around a definitive list of quality indicators. The list, arrived 
at by consensus, must represent the collective wisdom of a diverse group of adult education 
practitioners, rcsearchei*. students, and community representatives. This paper presents one 
practitioner's view. 

•Die author does not attempt to define sundards retative to the indicators except as 
necessary for tiie sake of clarity. In particular, when indicators are non-specific, as in "recruitment 
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methods* or *entiy policies,* stinditds are referenced as a means of refining or more specifically 
detailing the indicatcm. This paper suggests a rt*positioning of some of the indicators on the list 
developed fay Pelavin Associates, Ina a more specific detailing of some of the indicators, and the 
addition of one new indicator, Institutional FrameworL In addition, a recommendation is made 
that several indicators reference the specific accommodations necessary to provide equal access to 
adult learners with disabilities. 

PPnr.PAM CONTEXT INDICATORS 
1. Ingrttiitkinal Fram€work 

Most adult secondaiy education programs are administered within the context of a larger 
educational institution, whether a public school system or a community college. Unlike traditional 
K*12 instructional programs which are the primary focus of public school q/stem, and unlike 
traditional credit-bearing courses ^ch are the primar/ focus of community colleges, adult 
secondary education programs often must struggle for institutional support With institutional 
support, adult education programs flourish; without it, adult education programs mmi direct a 
disproportionate amount of time and resources to their own survival It seems reasonable then to 
require that a delineation of quality indicators for adult secondaiy education programs begin with 
statements of the parent's institution's mission and goab in relation to adult secondaiy education. 
In addition, there must be evidence that the institution pays more than lip service to its 
commitment to adult secondary education, Le^ that the institution meets or exceeds the pre- 
defined standards. 

Therefore, under PROGRAM CONTEXT indicators, the first should be Institutional 
Framework: There is a dearly suted purpose for the presence and operation of the adult 
secondary education program at this educational institution. Suggested standards include the 
following: (a) One of the written goals of the educatk>nal institution defines its oominitinent to 
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saving adult students who do not h«ve • high school diploma; (b) The institution commits 
financial resources to the adult secondaiy education program beyond the federally required local 
match; (c) The educational institution has a formal organizational structure for conducting the 
adult secondary education program (this includes a program director whose primary 
responsibtUties arc in adult secondary education; (d) Administrative support services (including 
clerical support, fiscal management support, computer programming services, and telephone and 
mafl privflegcs) «e provided to the adult secondaiy education program by education institution; 
(e) All of the student services of the institution arc available to adult secondaiy education 
studentt (Le^ career planning, job referral, Ubrary, student center, guidance and counseling 
services); and (f) Appropriate space is available for intake, processing, and counseling of students, 
for group and individual instruction, for work area for each staff member, and for storage space 
for instructional supplies and equipment 
2. Need far Program Services 

For this particular indicator, there seems to be some overlap between PROGRAM 
CONTEXT and PROGRAM PROCESS, specifically Program Planning. Given that there are 
approximately 50 million adulu in the United Sutes who do not have a high school diploma, 
there can be littiff question concerning the nssi for program services. However, to plan 
appropriately for the delivery of services, the adult secondary education program director and suff 
members must be familiar witii the profile of their community; they must possess current 
information concerning the number and demographics of the target population, the education 
levels of the community, and the annual number of high school dropoutt in the community 
(compared witii the number of adult secondary education completers by age). This data, collected 
under Need for Program Services, becomes the foundation on which decisions about program 
planning are made. Because this dau is tied inextricably to the prooeu of program planning, a 
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minimum itondard for every Program Pluming indicator should be the exiitence of evidence that 
decisions have been made based on an understanding and knowledge of the community proffle. 
3. Pry Miration mtti Strnctni* «f ntHvenr Svstew 

The definition of standards for this indicator should include the number of student contact 
hours provided by the institution. Standards also should specify that Uie program deliveiy system 
adhere to an open entry/open exit poUcy and tiiat project locations and settings be community, 
based and distributed throughout the designated service area (Lc not concentrated at one 
primary facility). Standards should specify that adult secondary education ofiferings be diverse 
(Le., GED program, external high school diploma program, evening high school program), that 
instructional facilities be botii accessible to and suiuble for adult learners, tiiat the number of 
students on tiie waiting list to enroll be minimal, and tiiat all instructional facilities be accessible 
to physically challenged students. 
4. QwnicteristtM of PTticiiwmts 

In addition to number and demographics of participants by skill level, by program type, 
and by program setting, anotiier indicator which should be included is ti»e number of participants 
witii disabilities who aic served by tiie program. DaU collected should include tiie type of 
disabilities of participants by skilllevel, by program type, and by program setting. In keeping witii 
Uie Americans witii DisabiUties Act of 1990, indicators of program quaUty specifically shoukl 
include Services/Accommodations for Adult Learners witii Disabilities. Ibis can eitiier be a 
separate indicator under PRCX3Ri\M CXJNTENT, or it may more appropriately be interwoven 
tiuoughout PROGRAM CONTB^ PROGRAM PROCESS AND CONTENT, and 
PROGRAM OUTCOMES. 
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pB (>ffP^M PgOTESS AND rONTENT INDICATORS 
L PROGRAM PLANNING 

1. CommniiHv Innnt In Pn »r«iii DevelODmwit 

Tbe f undtrds for this indicator ihould specify not just the existence of in «dvisoiy board, 
but alsc that the board membership be leprcsenutive of the community, including minority 
community residents, students, and rcprcsenutives of social services agencies, business and 
industry, and local government A list of board members and their affiliations should be available; 
board meeting should be conducted on a regular basis, and supporting records and minutes of 
meeting should be avaiUble for comination. In addition, the advisory board's function should be 
to advise and assist iH adult secondary education programs (ix, GED, external high school 
diploma, and evening school) offered through the educational institution; this is preferable to the 
existence of a separate advisory board for each adult secondary program. 

2. CoonHnatioB Activities 

This is perhaps the most significant indicator of quality for ASE programs. In these days 
of shrinking budgets and resources, a healthy and effective ASE program wiU have esublished 
numerous linkages with other agencies to ensure the effective deliveiy of services for students. 
The existence of linkages afifords studentt the best choice of appropriate services while allowing 
provider agencies to specialize in functions that they perform best 

There is virtually no limit to the number and type of linkaga which can be forged to 
strengthen ASE programs. However, minimum standards shouU include evidence of coordination 
in the areas of fadUty use, guidance, job placement, day care, transportation, tutor training, and 
coordination of volunteers. In addition, there shouW be evidence that cooperative linkages exist 
with kKal business and industry as weU as with other sutc and federally funded programs which 
provide employment services, job development, and vocational training at the local level The 
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ASE piogram director's responsibilities should include community involvement Jind pubUc 
relations, and there should be evidence that program information is provided regularly to 
community organizations, agencies, and otiier public service providers through a variety of means, 

both oral and written. 

Among local ASE directors, however, there currently exists a high frustration level over 
tiie apparent absence of coordination among federal departments tixat administer programs for 
adult learners. For programs funded through the U.S. Department of Education, separate and 
distinct annual and long-range phms are written at tiw sute and local levels for adult basic 
education, vocational education, special education, and vocational rehabilitation. In addition, 
programs funded through the U.S. Department of Labor's Employment and Training 
Administration provide services to many of the same cUents. It appears somewhat misguided for 
evaluatots at the federal level to require linkages among local service providers when it is tiie 
sparse coordination at tht federal level that sets up the barriers met by local planners in 

attempting to forge linkages. Ncvertiieless, coordination activities remains one of the most 

important indicators of program quality. 

3. Written Opewttenal Plan 

Program directors often prepare operational plans because tiiey are required to by tiieir 

fimding sources, but after funding has been received, tiiey rarely refer to the plans. Awritten 

operational plan shouW be the guide which steers the course of ti>e ASE program, and a quality 

program will demonstrate itt commitment to foUowing the plan. 

Hie plan should contain a mission sutement which includes ibc phil&»phy, goals, and 

objectives of tiie ASE program; it shouM sute the relationship of tije ASE program to tije 

misskm of tiie educational institution which houses ti»e ASE program. The plan shouW describe 
Uie interface among various ASE deUvery systems (eg., GED, external high school diptoma, and 
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evening high school) and should detail coordination activities and arrangements wth other 
agencies in the community. Specific program goals and objectives should be written in terms of 
measurable outcomes (Le, products, program changes, student accomplishments). 

Students, faculty, other administrators of the educational institution, and members of the 
advisory committee should participate in tiie development of tiie plan. Once tiie plan has been 
developed, the program goals and objectives should be disseminated to program staff, 
administrators of the institution, and advisory board members. There should be evidence Uiat tiie 
plan is reviewed periodically by program staff and advisory committee members for tiie purpose of 
making adjustments to program activities so tiiat suted goals and objectives can be met 

All of tiie quality indicators v^rhich Pelavin Associates, Inc. has listed under PROGRAM 
CONTENT, CTAFF QUAUHCATIONS, and CURRICULUM AND MATERIALS become tiic 
•stuff" of tiie written operational plan. These are tiie "inputs" or ingredientt which go into tiie 
making of a quality program. The indicators listed under PROGRAM OUTCOMES become tiie 
evaluation measures used to determine tiie extent to which each objective of tiie plan has been 
met Witiiout a written operational plan, tiiere is no way for administrators, evsluators, and 
legisUtots to know tiie direction tiie program is headed and no way for tiiem to know if it reaches 
its destination. 

n. PROGRAM CONTENT 

1. Recniitinent 

Each of tiie indicators listed by Pelavin Associates, Inc. under PROGRAM CONTCNT is 
important and should remain on tiie final list Under Recruitment, tiiere shouW be evidence tiiat 
tiie dau collected under tiirough a community need assessment (under PROGRAM CONTEXT) 
are used to target tiie poputations to be recruited. Tliere also should be evidence tiiat tiie 
program's written operational plan details diverse recruitment strategics, and tiut tiie recruitment 
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plan involves dl relevant segments of the community, not just the learner population. There 
should be evidence that the program director coUects student feedback concerning the 
recruitment strategies which have proven most successful among various groups of participants. 
n,e data gathered under Characteristics of Participants (PROGRAM CONTCXT) should be 
analyzed to determine which groups within the target population are being successfully recruited. 
2. fmyram Intake Proctdpm 

Incoming assessment, conducted for all students at the time of initial enroUment into the 
program, should coUect both placement and diagnostic information about student skill levels as 
wcU as information about student needs and interests. Assessment procedures should be non- 
threatening to students; assessment instruments shouW have been de«gned and dcvetoped for use 
with adult Icamers. During tiie intake interviej*. cUents should be given clear information about 
the purpose of the ASE program, including instructional approaches to be used. Oienttalso 
should receive results of all assessment pioceduies used. This information will enable the client 
to make a more informed choice concerning the suitability of Uiis ASE program to hisAer specific 
needs. It also will be useful in tiie development of tije individual student's learning plan. The 
learning plan must be decided on mutually by the adult student and ti»e instructor(s), using Uie 
results of both formal and informal assessments. Results of all assessment and testing shouW be 
maintained as part of the student's file. 
3. Onyotiw Asse sMieat Methods 

There shouW be evidence that instructors view assessment of student progrett as an 
integral part of tiie teachingfleaming process and a valuable tool for improving not only student 
progress but also instructional metiwds. Studente regularly shouU have an opportunity to 
demonstrate subject matter proficiency tiirough mastery testing (progress checks). Resultt of 
mastery testing should be shared witii the learner and used by instructors to modify the 
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instnicUonal program. An important feature of the assessment qstem is for program 
administrators and teachers to recognize that no one accountability measure is appropriate for all 
studenu; therefore, a variety of instruments must be available. Tbc accountabiUty system must be 
driven be the individual learner's goals, and programs must be evaluated on how well those goals 

are being met 

4. Supnoit Services 

The provision of support services is a critical element in ASE programs, but it is not 
necessary that the services be provided directly by the ASE program. When strong linkages have 
been established with other agencies (see Coordination Activities under PRCXjRAM 
PLANNING), the resources of the ASE program can best be freed up to provide educational 
services while other agencies handle the transporUtion and child care needs of adult students. 

5. Erit and Follmr-mi Proctdnres 

Ideally, exit interviews and testing should be conducted, vibcn possible, as students 
complete the instructional program and before th^ leave the ASE program. The program also 
should have an esublished procedure for contacting dropouts. However, adult studentt do not 
always inform instructors that th^ are withdrawing from the program and program staff may not 
be able to contact students easily after the fact to conduct adt interviews and testing. Although it 
is important for evaluaton to recognize that not aU dropouts will be contacted, a measure of 
quality can be that a given percenuge of dropouts is contacted successfully. 

IIL STAFF QUAUnCATIONS 
AU the indicators listed by Pelavin Associates under STAFF QUALfflCATIONS are valid 
and should remain on the final list Details to be added include the following: that criteria for 
employing program personnel are available on request as well as descriptions of duties/ 
responsibilities of each ASE staff member, that duties of program staff are related primarily to 
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ASE .ctiviUe*; that the ASE program has a Elm for ataff development and that the plan 
addresses the training needs of newly hired instructors through orientation and prc-servicc 
sessions as weU a» the needs of aU instructors through on-going professional development 
activities (indu^g attendance at sute. regional, and national adult education conferences); and 
that the educational institution budgets funds for staff development activities for botii full-time 
and part-time staff In addition, the findings of program evaluations, combined wiUi staff self- 
assessments of needs, should be used to identify training priorities. Recent developmentt and 
trends in adult secondary education should be regularly disseminated to and discussed mth 
instructors and otiier staff members, including volunteers. 

Where possible, tiie ASE program director should conduct an in-dass observation of every 
instructor every year. K tiie instructional staff is too large for one person to condurt annual 
observations, the help of otiier program administrators (principals, lead teachers, etc) should be 
enlisted. Evaluation of staff members should include input ftom administrators, students, 
volunteers, as well as self^uatioas-conducted by each staff member. Student attendance and 
retention records also should be considered because tiiey arc indicators of how weU studentt* 
needs are being met Evaluation resultt should be used for setting training priorities for tiie next 
year, tiiereby perpetuating the cyde of continuing professional development for all staff members. 

Tbc current part-time sutus of tbc majority of adult secondary education instructors is a 
major barrier to tiie ultimate success of the program. Part-time employees often are deprived of 
job subility. benefiJs. and tiie professional devdopment opportunities afforded otiier professiontl 
educators. TWs remains a 'fact of life" in adult education, and programs should not be evaluated 
negatively for employing only part-time instructors. Utopia will have been adiieved when tiie 
funding earmarked for adult secondary education programs is adequate to allow program directors 
to offer full-time employment to professional adult educators. 
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IV. CURRICULUM AND MATERIALS 
Program effecUveness measures in the area of cuniculum and materials must center 
around the individual learner's goals and how well the ASE program meett those goals. Teachers 
often feel a conflict between being a true adult educator, ic^ a faciliutor of learning, and 
"teaching to the test" as in GED programs or teaching the prescribed curriculum as in evening 
high school programs. Adult studentt are not likely to persist in classes that fail to meet their 
expectations, to make learning meaningful, or to provide flenbility. Adults are more likely to be 
motivated to learn wbcn instruction is custom>taik)red to their unique interests and needs. 

In ASE programs, however, wbxan academic achievement is one of the desired outcomes, 
there is a delicate balance which must be maintained between traditional teaching methods 
faciliution of learning, between externally imposed requirements of the ASE curriculum and the 
adult learner's need to be self-directing. This means that the learning objectives and e3q>eriences 
selected for the individual student's learning plan should be ones which will help the learner 
attnin some personal goals. These goals may match some of the goals imposed by the ASE 
curriculum, but th^ need not match them alL 

It also means that instruction which addresses a variety of learning styles and preferences 
must be provided, incinHing large and small group instruction as weU as individual or tutorial 
activities. There are some 20 to 25 instructional methods wfaidi can be empk)yed. These range 
firom buzz sessions to case studies to demonstrations to interviews to role-plays to fieki trips. The 
choice shouM be determined by the nature of the subject matter and the indhidual learning style 
of the student A critical element to the success of instruction is the skill and knowledge of the 
teacher in employing various methods. 

All of the indicators in the categoiy of cuniculum and materials on the Pelavin Associates' 
list are appropriate and shouki remain on the final list Specific standards shouU include the 
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•vailability of equipment, teacheis, methods, aad materiak for physkally challenged and learning 
disabled students. This is another area in which the establishment of linkages with other service 
provider agencies can be of considerable help. Materials at all instructional leveU should be 
available in suf&dent quantity for classroom use by all students, and materials should be up-to- 
date, free of sexual and cultural bias, and multilingual/multicultural, if necessary. There should be 
evidence that audiovisual equipment is regularly used as part of instruction and that computer- 
assisted instruction is avaflable to learners. 

PROGRAM OITFCOMES 
Evaluation efforts in adult secondary education programs traditional^ have focused more 
on process evaluation (inpuu) than on product evaluation (outcomes). For adult secondary 
education programs to have creoibility, evaluation must center around accounubility and program 
outcomes. Of course, legislators and funding agents want to hear that studenu vibo have 
participated in ASE programs have attained new employment, improved current employment, 
improved income, or attained employment-related skills. However, this ii not a faulty yardstick 
used for measuring ASE programs. 

If ASE programs teach academic skills and QQi employment skills, the measure of program 
effectiveness should be based on the studenu' acquisition of academic skills and the attainment of 
adult secondary high school diplomas, QQt on the attainment of employment and improved 
income. While there can be litUe doubt that these are indirect beneGts of participation in ASE 
programs, quite possibly derived from the acquisition and improvement of academic skills, they arc 
not able to be measured directly from program inputs. Retention and participation rates, 
educational gains, and goal achievement arc direct measures of program outcomes. Information 
about student employment gains is important for information for program administrators, but it 
should be not considered a direct measure of prognon effectiveness. This information wouW be 
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useful in foUow-up studies which compsrc the cueer dcvetapaeat «nd caniiag power of ASE 
program graduates with high school dropouts and with holders of traditional high school diplomas. 
After all, if we do not judge traditional high schools on the careers and earning power of their 
students, why would we want to do this for adult secondary education programs? 

Evaluation standards should include the following: the ASE program collects information 
describing the extent to which students* objectives are achieved, and the artent to which student 
profidency has increased in academic skilb; and program achievements are disseminated to 

learners, counselors, advisors, staff; other public agencies, legislators, and the local community. 
Program directors should continuously monitor program outcomes, share this information with 
students, staff members, higher level administrators, and advisory board members to plan for 
improvements to the program. Studentt should be surveyed concerning their pgrwption? of the 
beneOts of participating in ASE programs. Survey questions should address positive life changes 
such as basic skill growtii, increased participation as citizens, increased family bcneflts, increased 
personal growtii in self-esteem and self-confidence, and economic gains. 
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OUAUTY INDICATORS OF ADULT EDUCATION PROGRAMS 
FOR LOCAL EDUCATION AGENCIES 

Connie Eichhom 
Omaha Public Schools 

Since one source of funding for adult education programs is federal funding, it is 
consistent that many indicators of program quality should fit any adult education service provider. 
The basic goals and objectives of any program must faU within the contort of the federal 
guidelines and federal legislation, yet each program should develop goals and objectives reflective 
of specific local needs to deliver the best possible quaUty of service. However, there may be some 
indicators which are more applicable to or perhaps more difficult to accomplish for LEA 
sponsored programs. 

From the list of quality indicators provided, an LEA might have difficulty in helping 
students find meaningful employment or continuing to higher education or training programs, 
retaining staff because of part-time status, evaluating part-time staff effectively, expanding projects 
to the workplace, and providing child care and/or transportation support services. An LEA might 
have easier and immediate access to dropout statistics and to lists of the dropouts, for comrenient 
student recruitment 

Most of the sample quality indicators arc essential ingredienu in any adult education program 
which provides service to the uneducated and/or undercducated. atcgories which seem to be 
underrepresented or neglected are program evaluation, student and volunteer recognition, 
institutional commitment, student retention activities, and progr«n desire to further the field of 
adult education. In the area of goal achievement, there are many more items which could be 
included, such as fulfillment of social service or parole requirement, diizenship, or meeting INS 
regulations. 

53 



59 



Tlie extent of fervk« » program CM offer and the extent to whi^ 
dients vdth the greatest needs may be dependent on the mission and philosophy of the sponsoring 
organization, ihe basic mission of an LEA is to serve the needs of the kindergarten through 
twelfth grade age resident of the district, with the early childhood component becoming stronger 
inmanyplaces. n« high school diploma may be considered a termination point for studentt in 
an LEA. rather than the LEA showing much concern for those 16 years and older who need basic 
skill improvement 

The LEA sponsorship of an adult education program may in some cases be a true 
commitment to serw undereducated and/or uneducated adulu in the community or may in some 
cases be considered a peripheral service of the district One quaUty indicator m the LEA 
sponsored program might be the extent to which tiie LEA provides additional revenue, materials. 
faciUties. personnel, and support services for tiie adult education program. 

Another indicator might be the result of what happens to the adult education program in 
times of severe budget constraints at tiie tocal leveL Is adult education one of tiie first places to 
k)ok for budget reductions? What would tiie school administration do to save tiie program of tiie 
school board really wanted to cut or completely eliminate tiie adult education program? 

Institutional commitinent, which should be considered as a quaUty indicator, becomes 
extremely important in tiie high stress time of possible budget reductions. It is also important m 
tiie pursuit of additional sute andAir federal dollars, as weU as in tiie expansion of program 
services or in tiie response to changing needs witiiin tiie community or area served. 

The number of projectt and type of grantt tiut may be submitted are dependent upon tiie 

phflosophy and tiie institutional support by tiie LEA. For example, workplace Uteracy programs 
may not be viewed LEA as an essential ingredient in tiie addt education program, ^ 
mission of tiie LEA is to serve tiie K-12 population tiirough high school completion. Is tiie 



54 



ERIC 



60 



education of homeless adults something that all local education agencies are willing to attempt? 
Are piograms designed to help the non-English speaking adult become proficient in English an 
objective of the LEA? b serving the institutionalized or criminal adult population supported by 
the LEA? b a GED program supported more heavily than an ESL or ABE program, possibly as 
an alternative to graduating more students within the school district? K these activities are not 
viewed as a component of the adult education program, there will probably not be much change 
for expansion of services. 

AU quality adult education programs must measure well in the defined areas of program 
content, program process and content, and program outcomes to truly serve the needs of the 
adult students. Every program provider should laiow the needs of the potential clientele served 
and how those needs were assessed. The program provider also roust know what types of 
students and the skill levd of students enrolled. The student demographia should reflect the 
diversity and the needs of the community or area served. If the demographics do not support the 
results of the needs assessment, more active recruitment or program evaluation may need to be 
done by the LEA. The needs assessment should be done regularly and shouW invoke a variety of 
sources, community input, empkjyer and workplace needs, census data (if available), ethnic or 
minority group input, and high school non-completion statistics. An LEA may not conduct a 
needs assessment on a consistent basis, but rather it may concentrate on serving those students 
wbo have recently dropped out of schooL This may be especially true if the LEA can recapture 
sute funds when students come back to complete a high school equivalenqr program before the 
age of 21. 

By the very nature of being an LEA, the public school sponsored adult education program 
should have immediate acce» to dropout information and statistics. Recruitment of this 
particular age group of potential adult education students should be relatively easy to da 
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The locations and «etting» of projects in an adult educatioa program sponsored by an LEA 
may be limited to the boundaries of the school district Classes may not be held in settings 
outside the district boundaries or may only be held in school fadUties. If the adult education 
classes must be held in school buildings the LEA should make appropriate adaptttions to the 
facilities for adult learners. Are the classrooms such that there are ttbles and chairs to be easily 
arranged for flexible instruction and grouping? Is the furniture physically comfortable for adult 
studentt? U there adequate storage for the adult materials, in the likely evcm that the space 
must be shared with other UEA classes and actwities? Is there a quiet room for testing and/or 
special tutoring? Do the adult students have access to aU resources in the facility, especially to 
the computers and appropriate software? U there a separate entrance available for the adult 
students? Utherc parking? U there a break room or area? These kinds of needs must be 
considered when the adult education program is housed within a traditional school fadUty. 

Sometimes advisory groups within an LEA appear to be made up of mainly personnel 
&om the LEA. Tlie advisory board for an adult education program shouU reflect the diversity of 
the community by having various ethnic groups, employers, social service agencies, students, 
volunteers, and teachers represented. Every adult education program provider should have 
established poUcies for membership on the advisory board, the role of the board, the length of 
time to be served, regular meeting times and places, recording minutes, and making 
recommendatioDS. 

Any adult education program should become a stronger program by collaborating with 

other entities to provide services to adult students. ChiW care and transporution are support 
services that inay be difiBcdt for the LEA to provider, but often agreements can be worired out 

with other agencies for these services. Limited types of counseling, such as educational and 
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career, maybe provided by the LEA, but more extensive counseling needs may be better served 
through cooperative arrangementt with outside groups. 

Adult education programs sponsored by an LEA should have dear definitions as to who 
can be served and how the student can receive service. Must the adult student be a resident of 
the district to attend dass? Must the adult student have been a former student of the distria to 
earn an LEA high school diploma upon successful GED test series completion? Can the program 
conduct outreadi activities outside the LEA boundaries? Who can be invoNed in the outreach 
activities, students, volunteers, instructors, community agencies? Are services available only 
during the traditional school year or is there flexibiUty in the sdieduling? Must the student cnroU 
at a specific time and location rather than the dass location to reccwe service. Arc all the special 
services of the LEA, such as career assessment or special education diagnosis, available to the 
adult students? Are there transUtors to hdp the non-English speaking? What are the LEA 
poUdes regarding the use of outside resource speakers, videotopes, etc.? Is there a polity that aU 
instructional materials must be approved by the governing board? 

Assessment procedure* for monitoring student progress and learning gains shoukl probably 
not be much different, regardless of the sponsoring institution for the adult education program. 
Each program should have established appropriate diagnostic tools, recording procedures of test 
results, and letesting times. Every program should also have esublished procedures to record 
student goals and a system of momtoring individual progress toward the goal, w^ 

file of student work and an individualized education plan. 

AD adult education programs should have a student foUow-up plan, whether to monitor 
program dropouts or program completers. If a plan ii in place to attempt to contact the dropout 
student, information may be gained fior program improvement FbUow-up on program completers 
piwides more dau about who has been empkjyed or promoted, who has fOBC on to higher 
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educatton, or who has been removed firom the wdfa^ Tbc oommunity college may be 

particularly interested in which students go on to further educational training programs. The 
information on student follow-up may be difficult for the LEA to obtain if the cooperation and 
coordination with other agencies and offices in the community have not been developed. 

An LEA may have different requirements for adult education instructional staff than the 
community coUegc Any instructor, whether part-time or full-time, may have to have current state 
teaching certification to be hired by an LEA. The instructor may have to be interviewed by both 
the personnel director and the adult education supervisor. As in any program, tiie instructor may 
be limited to a very specific number of hours so that part-time sutus with no benefitt is 
maintained. Does the part-time employee in the LEA have access to services such as media, 

photocopying, classroom supplies, etc.? 

Some LEAs may require that any and all employees must attend certain kinds of in-teivicc 
or staff development activities. The adult education instructor may ti»en have UtUe opportunity 
for input as to the type of staff development needed. Ideally, tiie adult education program has 
input &om tiie instructional staff as to ti»e type, time, location, frequency, etc of tiie inservice 
activities. If instructors in an LEA sponsored adult education program have responsibilities otiier 
than teaching adults, it may be difficult to schedule activities to ahwys include aU adult education 
instructoB. The policies of the LEA may determine whetiier Uie instructor receives 
compensation for staff development participation or for travel expenses incurred for special 



events. 



All adult education programs should include an instructor evaluation component The 
LEA may have very specific procedures for fuU-time staff evaluation, yet die adult education 
program may be altowed some flen^iUty in tiie process or perhaps be complet^^ 
evaluation, especially witii part-time instructors. Some of tiie fleability may result form the part- 
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time or full-time status of instnictois and whether instructional duties include more than adult 
education. Not only should the instructors be evaluated, so should any volunteer and support 
StafL 

The goals and objectives of any adult education program should dictate the type of 
cuniculum and instruction used. Each project vrithin the program may have topio which need to 
be stressed more than others, such as reading in a low level ABE class or dtizensW^ 
class. TlKsinstnictional techniques used in the dassshouW be appropriate for th^ 
particular setting: individualized, smaU group, whole group, one-to^ne tutoring, or independent 
study. 

The materials used in the pK)gram should be appropriate for student needs, abiUty levels, 
and interests, as well as be multicultural and nonsexisL Manipulatives, workbooks, software, 
individual practice sheets, audiocassettes. videocassettes, supplementary readmg materials, 
calculatort. and newspapers shouW be available for student use in any adult education program. 
As stated previously, the LEA may have policies tiut aU instnictional materials have 
«iopted or at least approved by tiie governing board. Every attempt should be made to include 
students, instnictors, and volunteer tutors in the ptocess of materials selection and materials 
evaluation. 

In addition to the dunensions listed for retention in program outcomes, effort and 
activities to retain students in the adult education program until iadKidual goal completion is 
reached shouW be an indicator of quaUty. Some programs appear to emphasize stiident 
recniittnent. yet do virtually nothing extra to try to retain students in the program. Some 

activities might include incentives for completing a specified number of class hours, such as 
earning a pocket dictionary or a calcuUtor upon 40 houis of attendance, the devctopment of a 



59 

65 



student alunmi group, a monthly student newsletter, or firee GED testing if started during a 
previously identified time. 

Another dimension within the student retention categoiy might be the activities done to 
recognize student achievement F6r example, a graduation ceremony might be held for those 
students who successfully complete the GED test series. Another form of recognition might be 
certificates of attendance and/or achievement to ABE and ESL students. Articles can be placed 
in the local newspaper of special accomplishments. Special events, such as hoUday parties or 
potlucks, can be fim and socially important to adult education students. Equally important is the 
recognition of the volunteer tutors and stafiL 

Another indicator of program quality is the percenuge of students who remain in the 
program until their individual goals have been met K a high percentage of students leave before 
goal completion, a serious look at the program content and instructional methodotogies should be 
made. Are students given the chance to let the program administrator know what they like or 
dislike about the program? Is this done only when someone leaves the program, or is it done on 
a routine basis? ^ 

One area that may be difficult for the LEA to do weU is to help those adult students who 
have completed the program find meaningful employment The community college and/or the 
CBO may have better connectk>ns within the community to help students with the job search. 
TTie LEA may not have a job placement office that is accessible to the adult students or may not 
believe that it is an objective of the school district to help the adult student find empkiyment 
Therefore, that is one dimension which may be more diCBcult for the LEA to successfully 
accomplish. 

Hie LEA may not have the resources to help the program completer continue on to 
higher education or to another training program. Tbe community college may have a vested 
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interest in getting the ABE/GED student enrolled in its continuing education courses. This is 
another dimension that may be difficult for the LEA, unless strong coordination and cooperation 
exist wth other, organizations which can adequately provide these services to the adult education 
student ^ 

Whfle most of the quality indicators presented appear to be fairfy generic for all providers 
of adult education service, some indicators may need to be added. One indicator that appears to 
be underrated is that of program evaluation. Ibe list has sta£f evaluation, materials evaluation, 
and staff development evaluation. Tbe issue of total program evaluation is not adequately 
addressed. Dimensions of this indicator would include who should be invoh«d, bow frequently, 
internal versus external, whether specific guidelines are established, who receives the report, and 
who is responsible to see that necessary changes are made. For an LEA, one concern might be 
whether the evaluation of the adult education program is done the same time as the rest of the 
district or completcfy separately. Another might be whether it should meet the same criteria as 
the school district during North Central Accreditation or similar procedures. 

Another indicator of program quality which has been omitted from this list is the 
willingness of the program administraton to become invohred in long term projects, such as 
research studies or special demonstration activities, to contribute to base field knowtedge in t^ult 
education. This ^pe of quality indicator can be done by any program and should not be any 
more difficult for the LEA sponsored program than others to accomplish. Are the program 
directors willing to help the sute department in the development of a new sute plan? Are staff 
willing to be invoked in a comparison study to determine which diagnostic tool appears to give 
the best comprehensive information? Ate instnicton willing to spend their own time to develop 
materials for individual or special needs students? Are the administrators willing to be tbe first 
program evaluated under a new sute evaluation process? Are the staff writing arttdet to 

61 



67 



contribute to professional jounuds, to other adult educators can learn &om the experiences? 
What is the program doing to contribute to the field of adult education? Are staff encouraged to 
join and become active in professional organizations. Is conference attendance and presenting 
promoted? 

The list of quality indicators for adult education programs is reasonably comprehensive, yet 
consideration should be given to adding a few of the previously mentioaed indicators and 
dimensions. A program which has already incorporated maiqr or all of the dimensions and 
elements listed should be a successful program, meeting the needs of many of the adult studentt 
in the service area. Other programs which do not have some of these dimensions in place should 
strive to do so. thereby improving the services to uneducated and/or undereducatcd adultt within 
the delivery area. 
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COMMUNITY COLLEGE DELIVERY OF ABE: 
MEASURING EFFECTIVENESS 



Donna Lane 
Director, Oregon Adult Education Program 

Unique System 

The community college as a setting for ABE is unique in its comprehensiveness, ite 
parallels to the mission of ABE, and its mechanisms for collecting comprehensive data. It is often 
positioned within sute government as the major training and development prorvider. 

Because community college systems vary greatly £rom stete to state, I will describe the 
delivciy system in Oregon and assume it can be aJ least partially generalized to other states' 
community colleges. In Oregon the system has sixteen colleges and provides services through 
contracts for communities that are not in community college districts. The colleges are recognized 
by the Governor, the Workforce QuaUty Council, the Stetc Department of Education, and other 
state agencies as the state's major provider of adult training and basic skills delivery. ABE is not 
a stand-alone system in Oregon. It is holistically integrated with other work and education 
programs. 

Students who enter Oregon community colleges lacking basic skills needed for success in 
vocational training programs or in lower division college classes are identified through placement 
tesu and encouraged to take devetopmental education classes. Most dcvetopmental education 
post-secondaiy remedial classes are in the areas of reading, writing, and computing. Study skills, 
using college resources, goal setting, and decision-making classes are also offered. Few of the 
colleges have J^ystems for sorting students into ABE or tuition-based post-secondary remedial 
classes other than by student intent If students come to the college specifically for basic skills 
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improvement, they tie placed in free ABE classes. If they come to cmoU in a coUege program 
but lack prerequisite skills, they are placed in the post-secondary remedial classes. 

Oregon adult Uteracy levels are %eU documented. In addition to doing a county district 
analysis of the undeieducatcd identified in the census, Oregon conducted a statewide Uteracy 
usessment in 1990. Also, a common instrument is used for intake by all state agencies delivering 
basic skills. 

The Oregon Uteracy Survey, developed by the Educational Testing Service for a U.S. 
Department of Education assessment of the Uteracy skiUs of young adults and adapted by tiie U.S. 
Department of Ubor. was used to sample 2,000 Oregonians in five regions and was stratified to 
provide a representative geographic distribution for adults ages 16-65. Prose, document and 
quantitative Uteracy were assessed, giving Oregon agencies data from which to set adult Uteracy 
goals for the next two decades. 

Oregon has identified numerous benchmarks for improving its quaUty of Ufe in the next 
twodecades. Huee of the ten key benchmarks identified by Oregon's Workforce QuaUty CouncU 
focus on adult Uteracy. Hie State ABE Director in the Office of Community CoUcge Services 
(OCCS) is leading tiie development of a strategic plan to meet tiie Uteracy benchmarks. 

The statewide assessment used for aU cUents entering Oregon's correctional institutions, 
welfare reform programs, JTPA youth programs, and community coUege ABE programs gives us 
even more data for program planning. OCCS coordinates this assessment service. 

Oregon .has a statewide planning CouncO for Adult Education and Literacy thai advises 
the OCCS ABE stafiL It is composed of media, business, agency, program, community-based 
organization, corrections education, university, otiier research organizations, and student 
representation. Tht councfl has been most successful in elevating the visibiUty of adult Uteracy 
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programs in the state, in supporting the efforts of training, and in supporting the statewide 
Literacy Line. 

ABE programs, sponsored by community colleges (not unlike programs in other state 
systems), are offered on campuses, in learning ctnteis, in elementary and secondary schools, in 
soup IdtAens, in jails, in state corrections facilities, in Ubraries, at community-based organizations, 
via television satellites, and at other locations. 

There are few community-based organizations dircctiy ofifcring literaqr services in Oregon. 
Most subcontract throu^ community coUeges because pooling of resources provides more 
comprehensive services. Oregon Uteracy, Inc. (OU) is tiic major exception. It provides 
volunteer tutoring tiiroughout the state. Some local OU chapters have specific agreementt witii 
their load community colleges for joint training and 6ee tutor and student materials. The ABE 
curriculum and stafif development specialist has worked with OU and community college tutor 
trainers to develop joint training materials, to csublish seven training regions in tiie state, and to 
tram three master trainers in each region. Certification and training standards have been 
developed. 

Because the community college literacy services have been in place over twenty years, 
many referral processes and joint planning efforts arc in place. Most colleges are leaders of tiie 
local literacy coalitions, including all organizations in their communities that provide services to 
literacy students. 

Local college districts develop plans to meet the requirements of Uie Oregon Sute Plan. 
The plans also describe advisory committees, local coordiiiation. and recruitment and retention 
procedures. 

Centralized recruitment is provided by the Literacy Line which is widely advertised on 
television and radio, and in the newspapers. The welfare division regularly places Uteracy Line 
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stuffeis in food stamp mailmgs. Hus has been moct effective in increasing our services to those 
greatest in need The Uteracy Line connectt studentt and volunteers with local programs. 
Exceptional media cooperation with the Uteracy Line has contributed to program growth. 
Uteracy Line regularly compQes and publishes sources of student referrals. Currently newspapers 
(27%). television (21%), foUowed by agencies, and individuals arc the major program referral 



sources. 



Community coUeges publish dass information throughout their districte each term. In 
addition, colleges' pubUcations and marketing departments provide considerable recruitment 

assistance at the iocal level 

The community coUege system and its partner agencies face major challenges if they are to 
meet the benchmarks for a more literate Oregon. However, through joint planning efforts and 
new and emerging information about how adults learn, the coUeges and their community partners 
are already reporting increased successes in retention and completion of skill levels. A small, 
select survey of Brst year welfare reform participants indicated increased employment and 
incremental salary improvements after participatica. Programs are beginnmg to report post-test 
and other student achievementt that indicate that co-phmning. case management, nondupUcation 
of services, shared cUent proffles. shared demographic information, managed enroUments, 
intensive oricnution, provision of multimodal learning opportunities, contextual learning, and 
on-going monitoring and certification of student progress can both accelerate and broaden student 
learning. 

Appropriate Indicators 

In Oregon, attention is given to keeping students in programs long enough to complete 
one or more levels of instruction. We are finding out that students can make signiGcant gains in 
their basic skills levels in as Uttie as thirty houn. Knowledge about average gains of studentt 
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enables staff to individualize retention goals as weU as educational goals. Retention goals can be 
established with students at intake. Programs have documented student retention gains that staff 
members attribute to improved initial orienution, barrier removal, and goal setting. 

Many programs are now giving basic skills certificates at CASAS (California Adult Skills 
Assessment System) Levels A, B, and C They are integrating life skills and emplqyabiUty 
Level D skills into GED preparation. Providing students witfi benchmarks at the three levels 
below GED has added significance to Level I instruction and appears to impact retention. 
MontiUy award ceremonies provide extra socialization opportunities for students. TTieESL 
Curriculum Committee is currentiy piloting assessment instruments and hopes to standardize ESL 
level attainment 

We feel a need in our community college system to record student progress in several 
dimensions. Stratifying students according to ESL at three levels, ABE at two levels, and at adult 
secondary level as required in our federal report is helpful However, because of our CASAS 
system, we prefer to divide ABE into three levcb and would like to report GED and Adult High 
School Completion separately. We would like to look at program outcomes by student goals. We 
wouU report tiirce groups of students separately, tiiose who want to get a job; those who want to 
go on to higher education; and Uiose who want to gain personal outcomes. The three groups 
typically have different outcomes, require different curricula and have different attendance 
patterns. 

We add several items to our annual report because they relate to Oregon ABE goals: 
number of students receiving tutorial services, diversity of staff as reUted to target population, 
and growth in the number of full-time instructors. Staff and student incluswn in planning all k)cal 
and sute activities, staff devctopment needs assessments, and regular evaluations of the sute staff 
by kjcal program personnel are important components of our on-going quality checks. We 
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systematically evaluate aU staff development offerings. Community coUege staff members with 
curriculum and training expertise have been tra^ 
program evaluations. 

Some program and student indicators being used for all Oregon community coUege programs are 
as follows: 

Student demand; 

Job placement or transfer into additional training; 

Instructional cost effectiveness; 

Adequate services in place to serve volume of students; 

Facilities available; 

Equipment/supplies avaflable; 

Revenue projections; 

Course and program retention; 

Student success by offering {ABE, E5U GED); 

Success in subsequent programs; 

Completion of individual education plan; 

Staff expertise; 

Quality of the curriculum; 

Program's service to other disciplines; 

Program's service to the community; and 

Other. 

In our ABE evaluations we also examine administrative support of the program, including 
adequacy of budget as compared to other coUege budgets. Rank and tide of the program 
manager also appears to have some significance. 
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Problems mnd Issues 

Community coUege ABE programs arc part of the larger statewide education and training 
system that is also struggling with measurement and quality issues. ABE program personnel 
would hope the same type of program quaUty information they provide their administrators would 
be those required in federal reports and that the number of indicators provided at the federal 
level would be fewer and more general limited to indicators that are easUy interpreted in a macro 
setting. 

There are several measures of ABE quality cunently in use that are useless and 
sometimes harmful for measuring the impact of ABE Some are as follows: 

A. Percentage measures that compare number of students served in 
ABE programs against the census to determine a state's effort are 
erroneous unless there is a mechanism for collecting total state 
effort beyond ABE ABE count doesn't pick up all the Level I 
studenu served by other federal* statv, and local agencies or in 
college post«4Becondary remedial classes that are tuition^based The 
Umited funding does not begin to provide extended services; thus, 
reporting ABE numbers served as the number being reached is 
inaccurate. 

B. Measures such as obtained driver's license, entered another training 
system, gained employment, removed from public assistance, and 
even obtained a GED are underreported by students and teachers; 
and, they distort outcomes. 

K accuracy of the above information is important and is to be used in comparing and/or 
evaluating programs, then complex systems need to be devised to gather the information, such as 
agreements with Departments of TransportJ^cion and Employment and with GED Testing Centers. 
Within community colleges we have good access to computers and management systems. 
However, it is very difficult for us to report on outcomes that occur ouuide of our programs and 
that students "sometimes* self-report 

Dau currently collected by programs for federal reports on whether students attain 
employment, improve employment, and improve income arc both inaccurate and an unreasonable 
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acpcctaUon. It isn't icalistic to expect that . program i«>uld have subsequent employment 
iaformaUon or that a student completing a basic skills piogram has been trained for most living 
wagejobs. Ouradult education pTogramsmustbetiedtomA.ervices.emplcvment^^^ 
services, and additional educational opportunities. ABE progr«ns can coUect information about 
knowledge «id skills g«ned by studenu in the piognun. They should al«> periodically work with 
other sute agencies to foUow.up on students who receive state and federal education and 
training service, to determine tong term outcome, and which progr«ns and senrices contributed to 
those outcomes. 

FbUow-up information should also addre« family changes rculting from educational gains: 
aaivities such as driving, voting, participating in community activities, and leadership activities if 
students attribute these successes to their training programs. These must be gathered through a 
follow.up or a s«npling process. If they are only partially gathered through student self-reporting. 
they should be merely listed as "other- accomplishments, not quantified. 
Summarr 

-Whfle more eUte institutions may define exceUence as exclusion, community coUeges have 
sought exceUence in service to many.' (Buflding Communities. 1988) 

Community coUeges have filled the niche in America for serving community members who 
dropped out of high «:hool, who completed high school without essential skills needed to compete 
the workforce or higher education, and newcomers to America who need English skills. They 
often described as the coUeges for everyone, democracy's college., and open door coUeges. 
Most have the open access value imbedded in their missions. The mission and environment of 
community coUege. make, them a comforuble home for programs designed to assist adults who 
are in the process of improving their basic skiUs. 
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There are many advantages in the community coUege deliveiy system. Counseling, access to 
training programs, child care ccnteis, availabiKty of local transporution, handicapped access, 
personnel services, research departments, audited business services, access to technology, public 
information offices, staff evaluation processes, and computerized record keeping are just a few of 
the many benefits. 

Working within the Office of Community CoUeges Services, the State ABE Director not 
only has direct contact with kxal program directors but also with other college administrators and 
state training stafE. More important are the advantages to the student Basic skills education is 
not an isolated educational experience. Program outcome wiU be shortlived unless they are 
coupled with access to parenting education, other college classes, social interaction with peers, 
vocational training, and, especially, access to career counseling. Hie student goal setting in a 
community college program can be much more comprehensive and informed because staff and 
students are interacting with other programs, stafiE, and students. 

Our community colleges and our state partners are increasingly convinced that federal 
agencies and our state agencies can streamline and improve all program quaUty measures by 
reducing overlapping documentation, eliminating meaningless information, using common 
definitions across programs, and, yes, by integrated planning at federal, sutc, and local levels. We 
must report outcomes, not eEforL We have to have programs tiiat are easy to describe, easy to 
defend, and that taxpayers can understand and appreciate. 

Once we esublish our common definitions, we can ask the following service-related 
questions: What is tiie purpose of ti»e service? Who is the targeted group? What senices are 
provided? Do tiie services aUreUte to tiie purpose of tiieAct(s)? What group or groups provide 
the services? Are tiie sendees important? What difference do they make to individuals? Aretiie 
targeted groups actually receiving tiie services and benefiting from tiiem? What is tiie cost per 
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cUent and per lervice? Are the appropriate groups providing the services? Are we coUecting 
enough information to teU our story without providing superfluous or unnecessary information? 
What arc the long range impUcations for clients? How would our cUents improve tiie services? 

Dau coUection is not an issue in a str te tiiat has the community coUege as its provider. The 
mechanisms for coUecting the data are in place. However, the ABE data needs to be looked at as 
a data segment that must be combined wiUi Department of Labor and Department of Health and 
Human Services data to teU the big story. ABE must be recognized as the first importam step 
toward cHent success, but it should not be considered or evaluated as a job placement program. 
Nor should date available to a program while a student is attending be considered the complete 
Gle of what occurred to students participating in ABE programs. Periodic foUow-up sampling is 
necessary to get more accurate information. This foUow-wp could be done at either the state or 
federal level 
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A NEW FRAMEWORK FOR ASSESSING PROGRAM QUALITY: 
MEETING THE CHALLENGE OF THE NATIONAL LITERACY ACT OF 1991 



Sondra G. Stein, Ph.D. 
Consultant, Association for Community Based Education 



This year begins a great adventure for aU of us involved in adult Uteracy and basic skills 
education - as learners, practitioners, administrators, researchers and policy makers. The passage 
of the National Uteracy Act of 1991 provides us with the tools and the Congressional and 
Executive authority to do what we have long dreamed of doing: to create the infrastructure for a 
system that wiU: 1) embrace the diversity of our service delivery network; 2) enhance the quality 
of Uteracy programs and the capacity of literacy practitioners through research, training, technical 
assistance and evaluation, and 3) strengthen the links between us to so we can learn from each 
other and share with each other^^ we can see ourseWes as parts of one system dedicated to 
enabling every aduh - but especially educationally disadvantaged adults - to have access to the 
best quality literal and basic skills education. 

At this ircsient. when we are beginning to frame this system - to build a National 
Institute for Literacy, to create strong interagency Unkages at the national, state and local level, 
and to establish systems for program quality - it is especially important to community-based 
providers of Uteracy services that they arc included, officially recognized by Congress, by National 
leaders, and by sute directors of adult education as a critical part of the literacy delivery system in 
every state across this country. WhUe much of the attention to date has focused on what that 
inclusion means in terms of access to new sources of funding for CBOs, at ACBE we are as 
concerned with inclusion of CBOs and other nontraditional providers in the system development 
process. It is not that we minimize the value of the Uteracy Act's guarantee of access to the one 
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source of funding for adult Uteracy and basic skills that is stable, especially at this time when 
budget deGcits are drying up local and state funds for adult basic skills. It is simply that it is 
foolish to believe that the promise of "direct and cquitoble accas to AEA funds" for CBOs wiU 
actually lead to "direct and equiuble jjge of AEA funds" unless CBOs also have a strong voice in 
shaping the mechanisms that control access to those funds- The only way our new system can be 
truly inclusive is if the quaUty control and resource dissemination mechanisms created for it are 
framed to include CBOs and other non-traditional providers in terms of their values, assumptions 
and deCnitions. 

It is with this understanding that ACBE is pleased to participate in what we see as a 
project of pivotol importance in the implementation of the National Uteracy Act TTie Act calls 
for the development and implementation of indicators of program quality "to be used to evaluate 
programs assisted under this Utle...to determine whether such programs are effective, including 
whether such programs are successfully recruiting, retaining and improving the literacy skills of the 
individuals served in such programs." In other words, these "indicators of program quality" wiU 
essentially be the "gatekeepers" for AduU Education Act funding, once they are implemented by 
State directors of education in July, 1993. 

As part of their contract with the US Department of Education to assist in the 
development of these indicators, Pclavin Associates asked ACBE to review a set of preliminary 
indicators and to respond to them from the perspective of CBO providers by addressing the 
following issues: 

n Are there any unique aspects of CBO providers that should be taken into account 
in the development of indicators? What adjustments and adaptations to a genenc 
set of indicators may be necessary? 

2) Arc these indicators appropriate for CBO providers? Are any unnecessary or 

should any be added? If so, specify which and the reasons the measure should be 
added or deleted. 
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3) What problems or other issues would confront CBOS when using these or similar 
indicators? 

The best way for us to answer these questions is to step back and ask - and answer - 

three questions of our own. 

1) What would indicators developed by CBps look like? 

2) How would they be simUar to/different from the indicators Pelavin has developed? 
What is the significance of these differences? 

3) What kind of changes would be necessaiy to the indicators proposed by Pelavin in 
order for them to be appropriate to CBOs? 

In answering these questions, one by one, my goal is first, to articulate the values, assumptions 
and deGnitions that shape CBO indicators of program quality. Second, I will compare these 
values, assumptions and definitions with those impUcit and cxpUcit in the framework developed by 
Pelavin for the U.S. Department of Education. Fmally, drawing on this comparison, I will 
propose a shift in the approach proposed by Pelavin so that the resulting quality assurance 
framework can be used both to evaluate and to improve the effectiveness of the fuU range of 
adult Uteracy and basic skills programs, including those funded through programs established in 
the Departments of Labor and Health and Human Services (e.g., JTPA, JOBS, Head Start) as 
weU as those funded through the Adult Education Act as amended by the National Uteracy Act 
of 1991. 

L A CBO View of Quality Assurance 

111 start by trying to answer my first question: If CBOs sat down to develop tbeir own set 
of indicators for program quality, what would th^ kwk like? TTiis is a question that the 
Association for Community Based Education has been interested in since its inception. From 
1980 to 1983 the Association worked with member programs 'to develop an approach to 
institutional quaUty assessment and to define specifications for self-study that would be relevant to 
the practice of community-based education.* The Standards of Performance for Community- 
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Educational Imtitutions that resulted from this effort were published with a M 
A^<^<p ynt Workbook aimed at helping CBOs assess the quality of their own performance. 
WhUe these standards did not focus specifically on literacy and basic skills programs, the next year 
ACBE participased in a project sponsored by the Dayton Hudson Foundation - B. Dalton 
National Literacy Initiative to develop a set of fiuidelines for Effrrtive Adult Uteracy Programs 
(Rainbow Research. 1984) that did focus on literacy programs - although, this time, not just 
CBOs. Since then ACBE has been working with member Uteracy programs to strengthen their 
own performance assessment systems and to improve the quality of their programs. We strongly 
believe that rigorous standards of program quality and performance work in the best interests of 
the participants of community-based programs. Building on our past efforts, we have begun to 
construct a framework for quality assurance for community-based literacy programs. 

Hiis past fall, at the annual conference. ACBE convened a Taskforce to participate in the 
development of this frameworL The participants were representatives of more than a dozen 
community based educational institutions: they included large and small organizations from both 
rural and urban communities. Some were predominantly literacy and basic skilU programs; some 
offered a much broader range of programs, including community development, housing and child 
care, to populations that were native speakers of English and native speakers of other languages. 
Tlicy represented, in short, the full range of programs ACBE includes in its definition of CBOs as 
free-standing, not-for-profit organizations that: 

a) , are sanctioned by the communities within which they operate and are 
accounuble to them; b) have a Board of Directors and suflf that 
subsuntially reflect the population being served; c) reach populations, 
primarily in low-income communities, that are not served well by traditional 
programs; d) use methods and materials that are relevant to local 
conditions and the experience of the participants; and e) integrate teaching 
basic skills with broader objectives of individual empowerment and 
community growth. 
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While this definition includes within it implicit standards of quality, our taskforce's daylong effort 
to define the key elements of our quality framework did not begin with existing documents. 
Instead, it began with a consideration of our students - the adults our programs were set up to 
serve. Participants were asked to prepare for the session by thinking about a particular adult who 
it was difficult for their organization to serve, for whatever reason. Participants were encouraged 
to take a long-term perspective: to think about what that adult had been like when he or she 
first came to their program; again, after six months; after one year; after two years; after five 
years. 

Our session began with four groups of practitioners sharing within their groups these 
profiles of "hard-to-scrvc- adults and their changing needs over time. The t jsk assigned to each 
group was to envision the key elements of a community-based Uteracy program that could, in fact, 
meet the needs of these adults and their communities. After a hard morning's work, each group 
shared their "key elements" with the rest, and then, for the rest of the day, we worked as a group- 
of-the-whole to combine the work of all four groups into a list of priority elements. 

What was on that combined list? We called our first clement "Environment." This is hovv 
we described it: 

Prograns need to provide a safe, supportive, challenging environment or climate 
that includes as essential ingredients: a) resptct for all participants; b) freedom to 
make misukes; c) someone to mentor everyone, staff and students alike; 
d) opportunities for growth and change; and e) opportunities to fill multiple roles 
(as both learner and teacher). 

Our second element focused on " rnmrnunitv-Building ." This element was described as foUows: 

Programs foUow effective community-building processes and practices, to foster 
community within the program (among students, among staff, between students 
and staff) and to create strong multi-directional Unkages with the larger community 
(including students* families and community organizations, institutions and 
resources that are critical to student survival and success). 
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Our third element was 7 ..m>.r^.entered Mnictures and practices." It was described: 

Learners are fuU partners in the conduct of the program, and share responsibility 
with staff, board and other members of the community for the growth and 
development of the program. Uameis participate in settmg long and short term 
S fbr the program and in evaluating the program's success m mectmg those 
coals. The program develops structures and systems to support learner 
&menC induding training for staff in l"d^«hip development techmqu«^ 
training for students in goal setting, planning and evaluation as well as trammg m 
working in teams and in leadership development." 

Hicre were other elements, too. of course. I elaborate these first three to give you a sense not 
only of what these community-based practitioners considered most important, but how. starting 
from the perspective of real learner needs, this group thought about program effectiveness. 

We aU felt good about the work we had accomplished: no one questioned the rightness 
or appropriateness of these key elements. It was not untU I got home and was trying to make 
sense of the work we had done (alone, in my office, with sheets of news print spread all around) 
that I was struck by how different these elements were from every other attempt to define 
program effectiveness I had been involved with - and there were quite a few during the six years 
I worked for the Commonwealth of Massachusetts. While I had not yet seen the evaluation 
framework developed by Pelavin Associates. I couldn't help thinking that the indicators developed 
to meet the requirements of the National Uteracy Act would be much more similar to these state- 
level efforts then to the work of the ACBE Taskforce. Was our ACBE Taskforce taking an 
approach that would be consonant with the federal effort? When I sent back the notes from our 
session to all the members of our taskforce. I shared with them my concerns, enclosed a copy of 
the elements produced by one of our Massachusetts efforts ( Prinriplrs for Fffmive Literacy and 
Ra«c Skills Programs) , and invited everyone to think about these issues. 

And that's what I did. too. I set about trying to understand how the two models were 
different, what the source of those differences was. and their significance. 
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n. Two Roads Diverged in a V/ood^ 

Tbere arc ccruin things that arc very clear. For example, where you start and the 
questions you ask determine the nature of the key elements you deOne. I have already said that 
the CBO practitioners in the ACBE Taskforce began with a focus on students. THey also began 
mth a problem-soWng focus. I had asked them to start by Jescribing not just any student, but a 
student their program had difficulty serving. In developing a set of key elements each group of 
practitioners was answering the following question: 

•What are the key elements for assuring that adults from the communiti« we work 
with will be able to participate in our educational programs until they achieve their 
individual or community goals?" 

The question that framed the inquiry for both the Massachusetts effort and the Pelavin 
Framework was very different Initiated in response to legislative action, one at the state the 
other at the federal level, botii are attempts to standardize across systems a common definition of 
what constitutes an effective literacy or basic skills program. The question being asked might be 
phrased as: 

-What key elements should state agencies/state directors of education look for in 
Uteracy and basic skills programs when attempting to determine if these programs 
are effective in assisting adults to achieve their goals?" 

The framework that Pelavin constructed in response to this question is included, for ease of 

comparison, in the box below. 

Hicre arc a number of ways one could characterize the differences between these two 
approaches. We could say that the unit of analysis for Pelavin is effective service delivciy 
whereas the unit of analysis for the CBO task force is meeting student/community need. Pelavin 
begins with the formal analytical grid provided by a state plan or request for proposal, the kind of 
framework that makes sense when one's task is to standardize and optimize existing efforts of 
state directors. 
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h Evaluation Framework and Sample Indicators developed for US DOE 



A. TOOORAM CONTEXT 

1. NMd for Frogram StfvloM 

A. Numbtr «nd DwnogrM^ «^ 
pppuiaflon 

B. Utmoy Uw» lf» oommunlty 

C. HIph •chool dfop-ouli In communlly 
a Employmenl-raIsM MA raMcte of 

oommurUiy 

Z Org«4zirtion and Stnietoro of Dolh^ 
Syotom 

A. Numbor o( prp}#cti 

a loc«lioniuid Mtlingt of projects 

CTypoof proj#cto 

3. ChoraelorMoo of Parlklpomi 

Numbor and d05i>ogr«phtca of pwttctpanli 

by: 

A. «kmiovo) 

B. program typo 

C. program atttino 

B. PROQRAll PROCESS AND CONTENT 
I PROQRAIf PLANNING 

1. Community input in Program 



A. ExiMnco of an Adviaoiy Board 

B. Program Holda PubUc Haaringa 
0. Uaa of a naada aaaaaamant 

D. Ofhar aouroaa oonauhad 

2. Coor Anatfon Adtvftlao 

A. Exictanca of ooordination arrangamantt 

B. Typa of ooordinatad activttiaa 

Z. Wrtttan Oparational Plan 
A^Exiatanoaof apian 

B. Spacrio program goala and objadiva 
oonalatant wfth alata plan 

C. Ptan davalopmant prooaat 

IL PROQRAM CONTENT 

1. Racniltniafrt 

A. Bacfidanan! mathoda 

B. Population Targatad 

C. Program Outraach and publidty 



2. Program mtakaPreoaduraa 
■■Ai Enliy policiaa 

B. Incoming Ataaaamart procaduroa 

C. Oavalopmant of Individual laamlng plana 

3. Ongoing AaaaaamantUathoda 

A. Prooaduraalor monitoring atudant progrM 
«nd laamlng galna 

B. Prooaduraa for monitoring atudant program 
tovwd.goala 

4. Support Sarvteaa 

A. Typa of aopport aaivlcaa offarad 

B. Adaquaey of aanricaa for moating atudant naad 

5. Extt and FoNowHip Prooaduraa 

A. Wathoda for contacting drop-outs 
a Bdtiman^iawt 

ni 8TAFP QUAUFICATIONS 

1. CtMraetarlatIca of Staff 

A. Oamographica 

B. Educational Background, cradantiaia, 
ai^arianoa 

CNumbarofatiff 
astiltratantion 

2. Staff n a ap ona lb ll tt laa 

A. Outiaaofatafr 

B. Staff oommltmant 

Cl Sta« Input Into prognun daaign 

3. Staff D a valopman t 

A. Bdttanoa of ataff davalopmant componant 

1. Whanprovidad 
^Contant 

3. Duriticn 

4. Saquanlial training 

5. Staff oompanaation for attandanca 

B. Evaluation of ataff davalopmant activltiat: 

t . Syalamatic naada aaaaaamant for contanS 

2. Evaluatton of aetMtiaa by alaH 

3. Staff paitlclpabon In davatopmarrt 

4. Uaa Of Volumaar Staff 
A. Dutiaa ofvolurttaara 
&Voluntaar training 
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Tl Evaluation Framework and Sample Indicators iwnr^ 



Pi. CUfmiCULUM AND IIATCroAlS 
1. Typ^etCttfrtculumindliirtni^ 

B. Topical •mphtiit 

C. lortniatoniatw*nlqu» 

3. R|in^dh^«ock)«co^ 

a Ct»riK*M«loi«l#qulpm«*: 

AppropHiii to mw* pfogram and laamar 

2. 



SuWWanl amount to maat program and 
laamar n aad a 

talartlnn and Evtfuatlofi of liatarlala and 
Equipmant 

IMhod uaad toaaiact and avaluala 
equipmant and malariala 



PROGRAM OUTCOMES 

1. Raianyon 

A, Houra of Inatnictlon laealvad 

Participation iilaa lor population aubgroupa 

2. Educatfonat Oalna 

A. Standardiia latt aoora galna 

B. Compatanclaa attainad 

3. Empteymant 

A. Attainad naw amploymant 

B. tanprovad currant amploymam 

C. Imprcvad Inooma 

a Ailalnad amptoymanl-ralatad akllls 

4. OoalAchlavamant 

A. Achlavad paiaonii goala lor participation 



m CBOs doo-t r«ny begiB «i.h . b>mc^i « their stating potot tor ideoU^B 
elements is the complerity of the live, of their le^ners. Urn. too. malces sense, gi«n that few of 
.he programs represented in .he tasliorce «re -planned" or laid out in response to an RFP. 
Rather, they came into existence in response «, some community need, and grew and changed 
organically, adding or changing elements in response to changing org»u^.ion.l and community 
needs. 

As . result, «e could say that the rules of coherence go«nnng these approaches arc 
different Since the components of the framework developed by Pelavin are derived &om . 
forma, analysis of insUtutionalixed service deliver,, the coherence of these W or paraUe. 
components is a given, a lopcal outcome of the deductive process that produced then. 
Conversely, the logic that drives the CBO process is inducth* The principle that enables a.e 
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seemingly disparate components CBOs identify to cohere as a system is that all play a critical role 
in addressing the full complexity of learners* lives. 

We can also compare the role of values in these two approaches. Striving for universality, 
Pelavin intentionally leaves out elemenu like goals, objectn/es and philosophic orientation that are 
the locus of value and can be presumed to vaiy from program to program and state to state. 
Working within a smaller, more congruent universe, the CBO taskforce assumes a commonality of 
values. They start with elements that are already value-laden, that we might describe as 
philosophic orientations as much as components, like "learner-centered structures and practices" 
and "community-building practices." The values explicit in these elements are the glue that holds 
together discrete functions grouped within components, as well as shaping the approach to more 
traditional elements like "planning," "personnel," staff development* -governance," "support 
systems," etc. 

Finally, while the CBOs are interested in constructing a formative evaluation framework 
they and other CBOs can use to improve their programs, Pelavin has been spcciGcally charged by 
the U.S. Department of Education to develop a framework for summative evaluation across 
programs and states. 

It would be possible to sum up these differences by describing them as the inevitable 
results of a top-down v. a bottom-up process and to conclude that while the CBO process may be 
well and good for program improvement, it just doesn't do the job when you arc trying to build a 
"system." While this is the current conventional wisdom on the significance of these differences, I 
don*t think any of us participating in this process would be satisfied with this conclusion^ It 
polarizes issues that aren*t polar opposites. On the one hand, it presumes that the only reason 
CBOs or any nontraditional providers would participate in developing an evaluation system is to 
undermine or neutralize the process - to minimize the impact (in terms of what they need to 
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document and report) on CBOs. On the other, this kind of -summing-up* assumes the approach 
represented by the Pelavin framework is the only credible approach to developing an evaluation 
system. As a result, it locks us into a scientific management approach to adult literacy program 
evaluation just at the point that a new way of thihking about program effectiveness is emerging. 
In the remaining sections of this paper. I wiU introduce some of the assumptions underlying this 
new way of thinking and to explore its appUcation to the process of developing indicators of 
program quality for the adult Uteracy and basic skUls system, 
in. The Road Not Yet Taken 

Those of us who have been involved in workplace education programs over the past few 
years have had direct ocperience with tiie "quaUty" or -continuous improvement- movemcm that is 
transforming how leading edge coavpanies think about productivity. As results documenting the 
efGcacy of tiiis approach come in from corporations around the world, American educators arc 
beginning to ask what the educational reform movement can learn from it Professor Jacob 

III. Sdentinc Management ys. QuaUty Management 

Stampen of the University of Wisconsin helped to open this avenue of exploration in an article on 
-Improving the Quality of Education: W Edwards Deming and Effective Schools.- Stampen 
begins his article by reminding us that -educational reforms often have developed from ideas bom 
in industry. In fact, the Geld of educational administration originated as an offshoot of tiie 
scientific management movement in the early 1900's.- Since Dcming's ideas are -heralded as 
nothing less than the third wave of the industrial revolution.- replacing the ideas of scientific 
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management that shaped the second wave. Stampen is interested in exploring the extent to which 
Deming's philosophy and tools might be used to produce a similar transformation in education. 
rContemporary Education Review. Winter 1984) 

Since Stampen's article first appeared there have been a number of efforts to rewrite 
Deming's "14 Points for QuaUty Improvement" as an expUcit guide to school reform. Several of 
these points seem to have direct applicabUity to our efforts to improve the quality of adult literacy 

and basic skills programs. 

n An organization is a "connected system." A problem in one part of the "system" 
impacte the whole. Therefore it is important to break down barriers between 
departments and functional divisions and to encourage collaborative problem- 
solving and long-term plaiming. 

2) What connects the members of an organization is a sense of common purpose and 
common goals. Strong leadership helps to build constancy of purpose and total 
invoWement (by staff, students, and community members) m contmuous 
improvement efforts. There is continuous education of all participants so they can 
fully participate in the process of improving the organization, so they can all do a 
better job. 

3) People work best in situations where barriers to "pride in workmanship" are 
removed; where "slogans, exhortations and targets" that create "adversaria, 
relationships" and build a "climate of fear" arc replaced by leaderehip and traimng 
in effective methods to reinforce creativity and build community. This is true tor 
both staff and students. 

4) The uhimate test of quaUty of service and quality of product is customer 
satisfaction. While students are the primary customers of education, they also can 
be seen as workers (engaged in the learning process/the producUon of knowledge 
and understanding). Other customers are those who rely on the quahty of the 
education process - most noubly, employers, family, and the community. 

IV. Charting a New Path 

We need the framework being developed by the U.S. Department of Education to 
evaluate literacy and basic skills programs to incorporate this new approach to achieving quality in 
organizations. Instead of working in the tradition of scientific management and focusing on 
discrete program functions, we need this new evaluation framework to adopt the quality assurance 
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approach expressed in the Deming principles above, and to focus on Uteracy and basic skills 
programs as interconnected s>^tems. This approach assumes that a defining element of program 
quality for every type of basic skills and Uteracy program - not just CBOs - is the development 
and articulation of a clear educational phUosophy. Broadly understood by staff, students and 
community, such a defining philosophy is a critical component of the organizational development - 
process, creating a constancy of purpose which gaWanizcs action. It builds a shared community of 
values among all members of the program community - staff, students and board, alike - that, in 
turn, produces an "environment of information and trust" that encourages problem-solving and 
facilitates learning, invobement and empowerment at every level Everyone works smarter, and 
the program outcomes reflect this constancy of purpose (Rhodes. Th ^- Sc jiool Administrator. 
10/90. 11/90). 

If the U.S. Department of Education were to move in this direction in developing its 
framework for evaluation it would be breaking new ground. So far. efforts to translate the 
Deming approach to the world of education have focused on school improvement rather than 
school evaluation. However, the quality movement does offer us two very prominent models for 
evaluation. The Oisi and most weU known is the Malcolm Baldridge National Quality Award, 
administered by the National Institute of Standards and Technology of the U.S. Department of 
Commerce. What is particularly interesting about the Baldridge Award, from the point of view of 
developing an evaluation framework for adult education and literacy, is that its purpose is to 
"promote awareness of quality as an increasingly important elemenMto promote] understanding 
of the requirements for quality excellence, and sharing of information on successful quality 
strategies and on the beneGts derived from implemenution of these strategics" (Baldridge 
AppUcation, 1991 p. 1). In other words, the intent of this competition, Uke the intent of Congress 
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in calling for development and implementction of a system of indicators of program quality, is to 
improve the quality of a broad range of organizations across the nation. 

To achieve this goal, the Baldridge award examination process incorporates several key 
features which could be replicated in the framework developed for the U.S. Department of 

Education. These include: 

n «B emphisis on being non-prescriptive. In the words of one examiner, the 

Baldridee examination is 'an audit framework, an encompassing set of categories^ 
that telb companies where, and in what ways they must demonstrate proficiency, 
without prescribing what techniques or methods a company must use to get there. 

2^ an emphwis on diagnosis as well >s evaluation. Companies that aren't ready to 
undergo examination can use the Baldridge framework to assess their own 

strengths and weaknesses. In addition to outlining a set of items to be examined 
within each category, the Baldridge framework mcludes a sconng system that looks 
at the approach a company uses, the extent to which that approach is deployed 
throughout the company, and the outcomes of that approach. 

3) an emphasis on 'quality system integration." as expressed in the key concepts or 
values that lie behind the categories examined. 

While these general approaches to creating a evaluation framework are relevant to an 

adult basic education system of program quality evaluation, the Baldridge Award's focus on the 

development and deployment of management systems to buUd quality performance means that the 

specific categories examined are not directly translatable to the evaluation of adult basic education 

programs. However, there is another quality award. The Shingo Prize for Excellence in 

Manufacturing, administered by the Utah State University College of Business, which is based on 

the same values and concepts but which, because it is focused more narrowly, provides a set of 

categories that we could translate quite easily to the world of adult basic education. Tliesc 

categories are outlined in brief, below: 

It seems to me that this framework for quality assurance can be easily adapted to the 
evaluation of adult education and literacy programs. Section by Section it can be transposed on 
the three major sections of the framework developed by Pelavin. enabling us to maintain the 
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IV. Catefori€S of The Shingo Prize for ExceDence ia Muutacliiriiis 
I. Strategic LMdmhlp.liivolvtRMnt and Suppo^ 
K MrngMMfltlMdMhip 

B. Einplov## liivo»v#nw< ^ 

C. Diwinwi ProoM, 0|Mr«bom and SuppoiVS«fvio« Improvvn^ 



formal structure laid out, whUe we shift the focus within each section and within each topic area 
identified so that our indicators reflect these system quality issues. 
V. Adaptations for a Quality Assurance Framework: Section H 



PROGRAM CONTENT 

UL STAFF QUAUnCATlONS 
1. ChAraeterMce of Staff 

A. D^mographlM 

B. EdtiealioMi background. cf*<5«ntiias. •xpttri«nc« 

C. Numbarof Staff 
0. StafTRatantfon 



EDUCATIONAL METHODS, SYSTEMS 
AND PROCESSES 

m. STAFF QUAUFICATIONS 

1. Staff Rau mtn iafit, SalacHon and Ratantton 

A!'S!!eiMit lor racruillog alaff with appcopriata 
damogiapWc baekgcooiKl and appropriala aducattenal 

teikgraundrOf«<iMaiala and aKparia^ 

B. Pfoeaaaaa for anwiriiHJ thai IndMduala aa»aclad 10 
ba on auff ranact appropriala damographtc 
background. 

& ProoMM* lor «nMMii« tndMdualt a^riMted to 
!)• on ottfl wfloet •ppfoprtrto •*ic*ioo«l b«*gKHind. 

crodonlM* and «9(p«rtaM*. 

D. PiooMMO and •truebNM for tnvolvins 
•ludM**, fcowd and •oinmonlly iBWTtbw hi Wring 



£. PmoMMO Md tbuelufM for ofMHiring an 
appcopiMa aliff: oludant ratio i* maintalRad. 
for vvafy of etaaa. 
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This process of adaptaUon would be minimal for Sections U and ffi. Section D of the 
Pelavin Framevrork, "PROGRAM PROCESS AND CONTENT might be sUghUy refocused to 
highUght "EDUCATIONAL METHODS. SYSTEMS AND PROCESSES." The emphasis would 
be on how the program does its work. It would look at the "methods, systems and processes" used 
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by the program in each component area for evidence that these "methods, systems and processes- 
are appropriate to meeting the goals of the program and the needs of the targeted student 
population. -Die kinds of indicatois proposed under Oirriculum and Matcrials/Materials and 
P ^.nnment Used is what I have in mind here. TTius, under ]U Staff Qualifications (1) would 
focus on '^t.fTP.^ittnent. S -^"^--'^ ^"t^ntion Processes' rather than on -Characteristics of 
Staff- While much of the actual data looked at might be similar, the indicators would, again, 
focus on how what the organization does is appropriate to creating a staff that meets the needs of 
the learners and the goals of the organization. 

Similarly, icframing the final Tpnnp AM OUTCOMES- section to focus on 
-MFASURED P TfF^^^NT^ ^ PROGRAM QUALITY. PROOP AM GOALS ANP 
T.F AT^NER ACHIEVEMENTS- might aUow programs to tie what has been accomplished more 
dirccUy and cxpUcitly to how they do what they do. To take just one component listed under 
-OUTCOMES- as an example let's look at "Retcniion.- Retention matters because unless 
students actually participate in educational activities they won't make educational gains or achieve 
other goals. So we want programs to document the number and percent of students who stay in 
the program long enough to achieve their goals (which may include transfer/referral to a more 
appropriate program). And we want programs to track or document how the methods, processes 
and systems they employ for recruitment, intake, assessment, instruction, etc impact on learner 
retention. That is, after aU, what qualifies these components as indicators of program quality: the 
fact that how we do them has an impact on learner achievement 

What I am suggesting here is that this section must look at the full range of program 
outcomes. TTxis includes an expansion of the list of learner achievements idemified in the Pelavin 
framework to include the range of learner achievements identified in ACBE's 1989 Evaluation of 
Community Based Literacy Programs. These achievements, now embraced under the rubric 
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•achieved personal gods for participation." include outcomes that have been identified as central 
to the Literacy Act's goals of impacting the intergenerational transfer of Uteracy as weU as the 
skills of the American workforce. They include: 1) festering chUdren's intellectual, social and 
academic development; 2) strengthening participation in community activities; 3) building skills in 
critical thinking and problem-soMng; and, as a pre-condition for these outcomes, 4) enhancing 
learner self-esteem; and 5) enhancing learner self-determmation. 

Looking at such outcomes for learners reminds us that programs also may have other 
goals for impact on the community that are measurable and should also be included in this final 
section. Such "program achievements" might include such outcomes as: 1) adding a satellite 
program in a public housing project or some other setting where the program can serve a 
population it is important to target; 2) working with a community health center to enhance 
community member's knowledge about nutrition, ATOS and other pubUc health issues. -TTie 
indicators here would be expressed in terms of the impact on community problems. 

This approach to program outcomes also includes measurable improvements in program 
quality planned and implemented with the intention of enhancing or faciUtating the achievement 
of specific learner and program goals. By focusing on "improvements" in aU these outcome areas 
our framework encourages looking at changes in program quality, program goals and learner 
achievements over time. It is acknowledging that buUding quality performance is a process. 

The greatest changes would be required to Section I "PROGRAM CONTEXT of the 
Pelavin Framework. This section would need to be refocused on the internal as opposed to the 
external context Like the Shingo and Baldridge Frameworks it would focus on "STT^TEGIC 
LEADERSHIP AND VISION, INVOLVEMENT (of the community and the learners as weU as 
staff) AND SUPPORT (Strurtures and Processes for Planning and Change)." An outiine of the 
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topics where we would want to identify program indicators for this section is included in the box 
on the next page. 

Tbe topics included here are suggestive rather than prescriptive. They include topics 
currently listed under Section H Program Planning of the Pelavin FrameworL I feel they belong 
in this first section which includes those processes, structures and systems that set the context for 
undersunding how the program goes about iu daily work of assisting adulu in meeting their 

educational needs. 

VI. Where Do We Go From Here 

Moving into new, uncharted territory is exciting but also a UtUe frightening. The National 
Uteracy Act, with its emphasis on creating a high quality adult Uteracy and basic skills system that 
can meet our nation's needs for a weU educated adult population, is an inviution to us to develop 
new tools and approaches that vwU enable us to map the full extent of this territory. The 
Program Quality Assurance Framework outlined in this paper provides a set of tools which we 
beUeve is weU-suited for this task. While we have much work to do to fdl out the outUne - to 
develop and test indicators and standards for every topic and component - there are clear 
guidelines for this work, implicit in the approach to qualiiy evaluation discussed above. These key 
concepts are: 

The .pproach is oon-prescriptlve. While it requires aduh Uteracy and basic skills 
programs to develop clarity about their values and philosophy as a key to effective 
organizational development, it does not prescribe a particular set of values or 
specific program processes and structures. 

2) The appnMch Is diagnostic It assumes that Uteracy and basic skills programs 
ought to be invoWed ma process of buUding program quality. TTirough our 
evaluation we uke "snapshou" of this process to assess how well the program is 
doing at a given time. However, we don't warn our indicators to be static; we warn 
them to focus on how not what, on "continuous improvement." 
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VL Adaptatioiis for Quali^ Assonace Fnaeworic Scctioa Oac 
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x\ The «ppro«±fti«sse$tatefi»tlon of components into • fystcm. lUt^ 

k»kinV«t program elements in isotation, the approach looks at ho^ the clement 
contributes to the system and evaluates iu impact on program goals and learner 
achievements. 

4^ T*e«ppn>«Ji«ti«a»«»cMto«eraatl8toclfc»« the touchstone for qi^ 

assvnuBce. If . program is having difBculty meeting its recrmtment and retcnuon 
coals, if learners are not making measurable gains in identified achievement areas 
ibca something is wrong. The program must k»k back at its gpab and kxjk 
carefully at it's educatkmal methods, systems and processes to determme v-oerc the 
probkin is and what kind of changes it can make to better serve the needs of adults 
and their communities. 

An approach to program evaluation that is based on these quaUty improvement principles 
wiU enable us to buUd a nationwide adult Uteracy system that is strong and flexible; tiiat not only 
meets its goals for today but is capable of adapting to changing needs as we move toward our 
national goal of an adult population that possesses the skills and knowledge necessary "to function 
effectively and to achieve \hc greatest possible opportunity in their work and in their lives." 
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DEDICATORS OF PROGRAM QUAUTV FOR VOLUNTEER 
LITERACY PROGRAMS 



Kevin G. Smith 
Literary Volunteers of New York Sute 

The primary beneBt of education is personal empowerment Empowered people enable 
social and economic development The highest levcU of personal empowerment occur when the 
greatest number of indwiduak are educated to meet existing social and economic opecuiions. 
ni«e eatpecutions are in a sutc of continuous change, yet each must be met The disparity 
between cxpecution and fiilfiUment. between assimilation and dependency, between sidll required 
and skill avaUable and between input and output is widening. This dynamic reUtionship 
determines whether there are high or low rates of literaqr. 

Until recently compulsory education to age 16 was suEBciently effective meeting the needs 
of society and the economy. Accelerated economic and social changes have led to higher levels of 
personal dissatisfaction for many adults. One factor influencing this dissatisfaction is the increase 
in information which necessiutes more advanced skills for processing that information. While 
there arc many positive resulting from the advent of the Information Age", one of the negatives 
is the widening gap it has created between Uwsc who have the skills to access, understand and 
utilize information, and those who don't 

Short-term, the easiest place to begin is to further educate those whose skills are closest 
to meeting needs, m approach has been an effective, interim solution. But as we have learned 
in the JTPA system among others, this approach, by design, must indicate arbitrary selection 
criteria creating a wider gap between those chosen to be served and those left unserved. Rather, 

a long-term, comprehensive approach is warranted which provides access and appropriate service 
to any and all aduk learners in need. 
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To develop a comprehensive ipproach to Uteracy seivicc provision we must define Uteracy 
skills standards for the 1990-s. and begin to project those skills which wiU be required in the 
decades ahead, to enable service providers to adjust their input and output This prerequisite 
mandates significant research, analysis and trend extrapolation. It also requires more sophisticated 
and formal rcUtionships between industry, Ubor, government and education. 

In a sense, enabling the appropriate education of future generations is an easier task Uian 
closing the current skills gap. since short-term solutions mandate quick. poUticallyexpedient fixes. 
As with the relationship between personal empowerment and social and economic weU-bcing. 
there is a causal reUtionship between those currentiy affected and those who wiU be affected in 
the future which cannot be ignored. Undereducated parents raise undereducated children who 
inherit their personal, social and economic problems. A comprehensive system must address tiiis 
relationship to avoid \mting ofi" an entire generation. A comprehensive system must be a 
reasonable blend of short, medium and long range solutions. 

Hie current system engages only a handful of service sectors which bring varying resources 
and capacities. These sectors include: school districts and vocationalAechnical institutions. CBOs, 
two and four-year institutions of higher education. Ubraries, corrections and volunteer Uteracy 
organizations. 

To develop a comprehensive system a great deal of additional resource is required. To 
hold the current system accounuble for outcomes it cannot achieve is foolhardy. To esublish a 
universal set of program quality indicators which compares these service sectors as equal in input 
and output would be t serious error. Each must be enhanced and judged according to its merit 
Achievable goals shouW be established for each. Every resource must be employed to begin to 
transform wbtt we have into what we need. 
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Fbr both short- «id long-tcnn solutions, one of the best resources available is volunteers. 
Volunteer liter«:yprogran« have. kmg«id positive history of serv^ Quality indicators fov 
volunteer Uter^y programs have been almost exclusively based on client satisfection as agreed 
upon by tutor and learner. Ho^. expecutions have ch«,ged. As a result this service sector, 
as most, needs the time «id resource to adjust to the new expectations. Research is needed to 
determine what the ament and future outcome standards for sodrf 

are and how and where vohmteer literacy prognuns contribute. Volunteer Uteracy progr«ns must 
have a voice in determining output and what benefit that output has in meeting the current and 
future sldUs needs. 

Based on this analysis, a set of program quaUty standards specific to volunteer Uterac>- 
programs should be established. Once standards are determined, volunteer Uteracy programs 
would have a benchmark upon which to be judged. 
PROGRAM CONTEXT 
Need for Procrun Servfce* 

Tbc need for program services in Uteracy can be determined through an analysis of census, 
survey andtor test results. While this dau can provide projections for long-rang plamiing. it's 
most accurate when viewed as a snapshot of current conditions. Any unforseen social or 
economic change can create the need for program service where the need dW 
Any dau which indicates that a significant percenuge of the poputation is or is at risk of being 
unable to engage in positive social or economic endeavor are strong indicators of the need for 
program services* 

Tbc responsibiUty to determine the need for program shouW not be weighted heavily in 
«sessing the program value. It is iUogical. inefficient and unreliable to have every kxal service 
provider determining the need for Uteracy programs. A comprehensive system should be b«:ked 
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by a master plan which assesses the current and future needs of the nation, xute and local 
community. Programs should be judged on their ability to develop programs needed rather than 

on their ability to determine that need. 

Volunteer Uteracy programs cut across demographic sodo^nomsc and contextual 
service boundaries. In addition to bemg particularly t^cU-suited to serving adulu with the to^ 
skills and self-esteem, volunteer programs have the fkadbiUty to serve as adjuncts for adults 
receiving classroom instruction punuing higher academic and/or employment related goals or to 
serve those who have attained a credential but not the stalls it represents. 
OifanizatioB ud Stractvre of Dclhrery System 

Since the one-to^ne tutorial model is the predominant delivery design for volunteer 
Uteracy programs, each dyad (tutor/learner pair) must be viewed as a project In this area, the 
program's abliity to mate a ttneturt whkli ■taimlxes logistical bwiieis for both the leariKr 
•■d the tutor while maintainins a consistent commoBkatioB and support aetirork is the critical 
indicator suggestic^ quaUty. An urban program with a high percenttge of suburban volunteers 
who are unable or unwilling to meet the needs of inner city learners is an indication and example 
of poor program planning. Again, service flexibiUty is primary. A service delivery system which is 
orfjanizcd and structured- which serves those most physically and programmatically hard to reach- 
is the goaL 

Another quality indicator is the level of coordination and integration between and among 
service providers. Creatmg this community network is the mutual responsibiUty of ail programs - 
volunteer literacy and others - and if evidence reveals effort but no result - volunteer Uteracy 
programs should not be judged negatively. Many programs complain that they don't have the 
time or resources to make the connections rwiuired to make community connecUons. Faflure to 
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do so Im direct, negative cxmscqueiKC for the tonicR 

yw^t<in£ program quality. 
Characteristics of Fartidpaats 

CoI!cctiiig.aiiaIyzmg and effecting program improve based on the number and 
demographics of participants by skiU level program type and program setting - is an important 
capacity for any program. One of the potential problems agencies encounter is that the dau 
coaected is used to limit or define the program. Evaluation is easiest when there arc a minimum 
number of variables. Ibe tendency is to reduce the number of variables thereby pigeonholing 
programs into specific, nanawiy defined service areas. Volunteer literacy programs may be most 
susccpu^le to an effort to limit piogran^i variables, for th^ have the greatest n^^ 

For example, the perception that volunteer Uteracy programs serve the hardest to reach, hardest 
to teach has Jed some sutes to delineate the popubtions to be served 1^ tto 
level odusivcly. LVA-NYS dau indicates that over 40% percent of those seeking service from 
Literacy Volunteers are employed and not in the assumed classification. The common threads 
seem to be program accessiT>iBty and choice based on the learner's perceived or real nce^ 

onc-to-onc, more confidential support 
PROGRAM PROCESS AND CONTENT: 
I. Prograa Plaaalag 

Comaaal^ lapat la Program Dcvckq^neat 

Any program designed to serve a community needs to utilize the resources of that 

community. Tim is especially true of a volunteer Uteracy program. National and sute volunteer 
programs offer communities a program model training and technic^ assistance to develop a 
service in their community. Sute agencies work to empower the community to address its own 
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needs. Tbe concentratkjn and focus b« been oo the oeed* of ii^ 

larger social or economic expecution. 

Volunteer Uteracy programs, as private, non-profit entities, must be operated bj- a Board 

of Directois- Tbesc Boards should be comprised of a crois-section of community members - 
including adult learners. Organization plans and policies should emerge through the deUberatk,n$ 
of these representatives. LVA-New York Sttte experience shows that local programs which 
include adult learners in the program and policy process arc stronger, more viable organizaUons. 

CoordiMtkm Activities 

If the goal is to develop a comprehensive continuum of service to meet the personal 
social and economic needs of each and every conmiunity then, dearly, the existence of 

coordination arrangements and the articulation of the types of coordinated activities are indicators 
of good program planning and program quality. Tbesc conditions most often occur in an 
environment characterized by a dear community, sute and national 

and dianges it wishes to effect, mutual program and professional respect and adequate resou^ 
-nxesc prerequisite conditions do not currentty exist, therefore, expectations must be adapted. 

It is right and good to foster coordination activities among programs to benefit learners 
who wish to transition and to share resource. However, as long «$ this is an underfunded 
enterprise that learners choose to engage, a program should not be judged on how wcU it 
coordinates with other programs- Rather, program quality should be judged on how wdl the 
Uteracy program screens it applicants to insure appropriate placement, iu 
knowledge of other services and its systems for getting that information to the people who need 
it, when th^ need it 



98 



104 



Written Opcntioaal Pfaui 

Adult contimung education is one of the most dynamic growth, issue industries of our 
time. Tbe enterprise is cunently characterized by an caving definition, a lack of adequate 
research and empirical dau upon »1uch to base prediction and decision, a nebulous sense of 
purpose, insufficient financial resource and a dearth of expertise. Good planning requires 
sufficient human, fiscal and information resource. Tberefioie, planning is very difficult 

Despite these barriers, all programs should plan. The planning process should be as 
thorough and inclusive as possible The plan should be written and be subject to constant revie*- 

and revisioo. 

LVA-NYS has found that some of the resources required arc available in the community, 
frequently through large corporations with planning personnel or the consulting firms utilized for 
this purpose. The national volunteer Uteracy networits, often through their mid-level support 
systems, provide planning support for their local programs andtor advise as to how and where to 
gctsupporL Evidence of the awareness of the need to plan or an effort to plan should be viewed 

as positive and built upon, 
n. ProgtvB Contest 
RecniltBcat 

Volunteer Uteracy programs have two constituencies in the community: volunteers and 
learners and must have solid recruitment systems for each. Learner recruitment is the priority. 
The most successful approach is a multi-layered direct and indirect strategy. Direct activities 
should include a fuU range of print, audio and video media. The most effective recruitment 
systems indude targeting the individuab who provide infonnatioo to those who a^ 
access it tbemsehes. Indirect measures are those which change policy and prejudices that limit 
program access. RenKJving the personal stigma which has long been associated with illiteracy is a 
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vety significant rccniiimcnt activity. AEfecting supportive change* ixt public polic>- is important A 
program's attention to learner advocacy sbouU be considered when 

Recruitment of volunteers is another, but equally essential, aspect of a good volunteer 
literacy program. Chiality programs only recruit and train enough volunteers to maintain 
of service required. Trained volunteers who are never matched arc wasted resource and a 
negative recruitment factor. If those individuals share, with even one other potential volunteer, a 
negative experience additional resource is tost. In this environment volunteers are difSculi to 
recruit and retain. Good programs understand their needs and control their rer^uitmcni activiti« 

to meet them. 

Programs that continue to recniit when they do not have the resource to serve and 
support should be viewed negatively. With the current national attention to the issue of adult 
iUiteracy. some programs have all they can do to keep up with externally stimuUted expccuiion. 
Good volunteer Uteracy programs have learner and volunteer recruitment procedures that they 
can adjust to meet need and capacity. Hie recruitment of program support should be an 
on>piing, cotisbtent effort. 
Prognm IiiUke Procedures 

Most volunteer Uteracy programs have an informal set of entry policies but would be hard 
pressed to produce them in writing. Tbe critical issues have to do with making dedsiora about 
whether the program can assist the individual and whether the learner has the right intentions for 
seeking assistance. Some programs, for example, have created entry policies wliich do not permit 
service to agency referrals which mandate partidpatioa Hie experience has been that learners 
who are externally motwated don't foltow through. Other programs have reported interview 
techniques which listen for ulterior mothre, such as. the male learner who specifically request a 
female tutor. 
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For the most part, programs tiy not to have policies which create categotical cxclusiotL 
Volunteer Kteracy programs, for example, ate one of the few resources available to serve adultt 
with learning problems. Many aduhs indicate that they have a kerning disability when it may be a 
problem based on other than clinical disfunctioii and can be addressed by a trained volunteer. 
The key to quality ts capacity of the program to match resource with need. Since volunteers 
come with vaiying Icveb of skill we cannot say that we are unable to serve any individual 

Most volunteer Uteraey programs utilize some type of diagnostic tool to dct ermine 
deficiencies in reading skills. Most do intake interviews designed to elicit personal gpals and 
expectations. Tutors arc trained to dcvetop individuallearning pUns fcom the information 
gathered during intake, from the diagnostic tool and through rapport buiWing techniques. 

One of the critical factors is that planning decisions for eadi learner must invoKe the 
learner; aU plans shouW be agreed to by the learner. Evidence of these characteristics is an 
indicatiou of high program quali^. 

Good programs ptovide their learners and volunteers with realistic cxpecutions. One of 
the most difficult challenges is to assist the Icariier who desires a GED and is reading at the 2nd 
grade level to accept the distance between these two points and not make it too daunting. 
There are typically no short cuts to this type of goal achievement. However, a program which 
fails to teach skills contextualized withm a desired specific personal, social or economic outcome 
articulated by the learner is risking Mure. Most leameis have veiy specific goals which made 
them decide to seek assistance. Programs which substitute goals that meet program, funding or 
management outcomes that are inconsistent with peisonal outcomes wiU negatively impact learner 
success and retention and not meet either set of outcomes. The best programs are those that 
hear what the learner wantt and needs and devetop learning plans which ptot the most direct 
course to achievement 



101 



OsfolBC AsscssMCBt Methods 

Most iearnen «nt to effect «wnc positive outcome or ch«»ge in ti«ir personal social or 
economic condition now «ul to improve their capacity to effea positive personal change in the 
future. Improved leading skilb are a universal prerequisite to this type of ^ They 
therefore be a primary approach to impro«d Uter«y skills. It is the opinion of this author that 
reading is only a part of successful information processing and an individual's ability to effect 
change, A learner. ^ can decode but not comprehend, y>ho can re«l but bas no knowledge of 
how to access relev«,t information, who possesses the needed information but doesn't know how 
to use or cjcplain it, is stiU dependent (or functionaliy illiterate). 

Procedures for monitoring student progress toward goal achievement should inform 
student progress and learning gains expected, monitored «k1 «:hieved. In other words, learning 
g«,« are the meam to «±ievement of personal social and economic goa^ They are the means 
to an end. If the means do not enable the desired end but some learning gains are being 
achieved, the program can be considered ineffective. Programs should offer leammg 
opportunities specific to the context in which the learner desires to effect positive change and be 
judged primarily on their capacity to enable goal achievement 

Some learning gains occur whfch are critical to goal achievement but diOkult to monitor, 
•mese gains arc in the life skilb andA.r affective domain. One of the greatest strengths of a 
volunteer tutor is that he/she is a information processor who employs hisAier skills to effect 
desired change in hisAier own life and that of their student In many respects, a voiunteer tutor 
.cts as model and mentor to the adult learner they assist Tbc volunteer tutor has the greatest 
Uutude in using the community as classroom. If «iult learner indicates difficulty in accessing a 
needed social program or making a purchase or any other personal social or economic function, 
the tutor can and should address that specific need and learning opportunity. By indicating ana 



102 



ERIC 



103 



showing their learner how they deal with simUar Ufe situaUons they are modeUng and teaching 
skills and addressing the immediate concern of the learner which, if not addressed. wiU probably 
inhibit learning. 

This overaU type of tutor^tudent activity should be considered literacy training. Yet 
funding requirementt to demonstrate grade level growUi in reading conflict with, and even inhibit 
this type of activity. The whole volunteer Uteracy system is set up to train and monitor tutors in 
reading instruction. Activity that doesn't feel like reading instruction is probably avoided. Yet, 
tutors have far more experience and training in proceuing information necessary to solve life 
problems than they do in leaching reading. When the defmition of literacy was solely based on 
the abiUty to read, the focus on reading instruction made ucnsc. With greater enlightenment as 
reflected in the definition of literacy in the National Uteracy Act the type of activities described 
are clearly appropriate, measurable, fundable and needed. 
Support Services 

In many respects the delivery design of volunteer Uteracy programs responds to the myriad 
of support services needed. In a one-to-one relationship there is greater flexibility to negotiate 
and mitigate logistical issues and to follow-up on referraU specific to the need of the individual 
learner. The critical factor for the volunteer literacy program is making sure that tutors and 
learners have access to the information. It is a significant responsibility and expenditure to gather 
and maintain a data base and information dissemination system on all related support services 
available in a given community. Again, that the program is aware of the importance of such a 
capacity and has made reasonable effort to achieve thU goal based on the resource available is an 
indication of program quality. 
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aearly. the relationship between support services offered or accessible through referral 
and the needs of the learner is critical. Attrition attributed to a lack of or inappropriate support 
services is an indication of program quality. 
Exit and FoIIow.up Procedures 

AM programs should distinguish between positive, neutral and negative termination. 
Positive termination include reasons such as goal attainment or transition to another, more 
appropriate service. An example of neutral termination is that the learner moves out of the 
service area or makes a personal decision to discontinue based on factors the program cannot 
control. Negative termination is that tiie program failed to meet the learners' needs and/or faUed 
to refer them to a more appropriate provider. 

A goal of all Uteracy programs should be a fuU range exit interview including final learning 
gam toward goal attainment evaluation. Volunteer Uteracy programs have a particularly difficult 
time performing this function and getting this data. To some extent, this is so because of a failure 
to emphasize the expectation to the tutor cadre and. to another extent, it is the result of the 
nature of the population being served. Many tutorial relationship end because tiie learner simply 
stops showing up. 

ra. STAFF QUALinCATIONS 
Characteristics of Staff 

Ideally, staff composition should reflect the population of learners to be served in a 
community. In some cases that balance existe. However, in most cases, the learner population 
base is multi-cultural, but the staffing patterns are not 

Dramatic changes have occurred in the volunteer literacy network during the past several 
years in the area of staffing. Not long ago few programs had the "luxury" of hiring paid staff; 
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those programs that did typically hired from within the Uteracy program ranks, and often to 
perform program specific tasks. 

However, that scenario has dramatically changed. Most volunteer literacy programs now 
have some amount of paid staft An equally dramatic change is occurring in who is being hired. 

Staff now have diverse educational, cultural and professional experiences and a wide range 
of credentials. Newly hired staff often have backgrounds in volunteer and/or not-for-profit 
management, fund development, and adult education among other fields. Although these 
professionals bri. g valuable and needed expertise to tiieir volunteer Uteracy program, tiiey often 
have limited, if any, specific volunteer literaq^ experience. 

Witiiin the LVA-NYS network of 50 programs, two have no paid staff, six to eight have 
one part-time paid staff peison. and the rest have between one and seven full-time paid staff 
positions. Without exception, all programs witiiin the network are undentaffed. 

Recruiting and retaining high quality professional staff is a major concern of volunteer 
literacy programs; salary is often not commensurate with other not-for-profits m tiie local 
community, and subsequently significant staff turnover cxisto. That is slowly changing as programs 
and the field begin to recognize tiiat, just as volunteers aren't free, neither are quality 
professionals to direct them. 
Suit Responsibilities 

Staff responsibilities range from Executive Directois having "hands on" direct 
programmatic and fiscal responsibiUties to hill or part-time tutor trainers, bookkeepers, office 
managers, clerical support staff, etc 

Volunteer Uteracy positions have responsibiUties simflar to those of other not-for-profit 
organizations (le., bookkeepeis, office managers, clerical support, etc.) TTieir duties, with few 
exceptions, are not unique to volunteer Uteraqr programs. 

105 

111 



f 



Positions and duties unique to volunteer Uteracy programs often involve training and 
program specific issues such as Tutor Trainers. Program Coordinators (responsible for volunteer 
and learner intake and support functions), etc. 

As indicated above, staff commitment ranges from having no paid staff or one part time 
Coordinator to programs having several fuU-time staff membe«. The constant that exists among 
volunteer Uteracy programs is that individuals often assume duties beyond their job descriptions 
and beyond their aUotted number of hours. It is not unheard of (though not encouraged nor 
recommended) for a Coordinator paid for four hours per day to work full time at their volunteer 
Uteracy position. Rarely is a lack of staff commitment an issue within volunteer Uteracy programs. 

Steff often designs, implements and evaluates programs; the fact is. if they didn't, 
innovative design would not occur. Being part of a national or sutewide volunteer literacy 
organization, much of the structure, training format and program design at the local level is 
prescribed or recommended. Given that foundation (i.e., provided wiUi an eighteen hour tutor 
training design) greater opportunity exists to design new or supplemental programs (i.e. family 
Uteracy programs, workplace Uteracy programs, etc) 
StcIT Devctopment 

A primary advantage of membership within a larger network or organization is tiie 
opportunity provided for suff development Training (pre and in-service) is often avaflable 
through mid-level (i.e., state level) or national organizations. Local volunteer Uteracy staff 
development issues exist throughout the network, unfortunately the resources to meet those 
needs are not adequate. Mid-lcvcl (whether regional or state) organizations that exist to provide 
technical assistance to direct service providers are often in the best position to develop and 
provide local staff development training. 



ERIC 



106 



II 



Staff development functions include organizational/volunteer not-for-proOt management 
(le., budgeting, finance, public relations, personnel, etc.), program specific issues, and adult 
education/Uteracy training. Because volunteer Uteracy programs are often not in a position to hire 
multiple staff to handle multiple rcsponsibiUties, one person is often responsible for many areas 
and needs to be trained and supported in alL Training should be sequential and ongoing. 

Evaluation of staff development activities arc needed throughout the field. Those mid- 
level organizations presently providing local staff development activities typically evaluate those 
activities informally and often merely through participation as opposed to planned transference of 
skills. 

Use ofVolnntecr Staff 

Volunteer suff is the lifeblood of any and all volunteer literacy organizations. Whether a 
small volunteer Uteracy program of less than 25 learners or a large program wiUi more than 1,000 
leameis. volunteers play a crucial role throughout and provide important services. Volunteers 
recruit, orient, train, assess and match learners and tutors. Volunteers direct and manage the 
support of other volunteeis in the programs. Without program volunteers (Le. tutors and trainers) 
and administrative volunteers (Lc, board members, office staff, matchers, recruiters, intake staff, 
etc) volunteer literacy programs could not cxist. 

AU volunteers need, and are entitied to, training regardless of their role in a volunteer 
Uteracy program. First and foremost, volunteer tutors should be provided with high quality 
training to enable them to maximize the tutoring experience for themscWes and for the learner. 
Poor tutor training is the precursor to poor tutoring. The volunteers that train tutors also need 
quality preparation and training. Without quaUty tutor trainers, a ripple effect of ineffective 
training and tutoring occur. Administrative volunteers need training as m«U: recruiters, matchers. 
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board members, testers, etc., are aU entitled to training in how to do the job they are being asked 

to do within the volunteer literacy program. 

EmluatioB of Staff Pcifomuuicc 

Relatively new to areas in the personnel field, volunteer Utera(y program staff evaluation 

practices are diverse, if present at all. Evaluation of staff performance is often informal and 

based upon observation. All areas related to personnel are developing within the field, with 

evaluation of staff among them. 

IV. CURRICULUM AND MATERIALS 

Type of Curriculum ud Instruction Used 

Hie history of the two national volunteer Uteracy programs differs principally in regard to 

the type of curriculum and instruction used. One has appUed an individualized, student goal 

oriented approach utfliang eclectic, often tutor made instructional materials while the other has 
employed a sequential, phonics-based instructional model The former assesses according to a 
pretest and post-test designed specifically for its use. TTie latter charts success along serial set of 
instructional materials. Both have been successful based on learner satisfaction. Both have 
evoh«d to include and encourage other techniques and instructional materials. The 
phonics-based program is undergoing the most dramatic shift in approach. 

As long as the abflity to read is a critical prerequisite to information processing, enhanced 
reading abiUly will be an indicator of learner and program success. However, it has been 
suggested throughout this document that volunteer Uteracy programs arc more well-suited to a 
curriculum based oa experiential learning. Tutors trained their whole lives to process information 
necessary to control and effea life change arc more wcU prepared to impart those skills than they 
arc to teach reading after an eighteen hour tutor training. TWs approach suggests that tutors 
build their separate curricula in consultation with their learners to address the immediate goal of 
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the learner. The tutor could then model their successful informaUon processing skills for the 
learner and act as mentor and advocate. Learner skills gaps would be exposed in a practical 
application assessment rather than through normed or criterion referenced test data. Curriculum 
and instruction used would be specific and practical 

Both national volunteer Uteracy organizations produce appropriate materials and review 
and recommend other potential instructional materials. Many local volunteer Uteracy programs 
have staff to assess and recommend appropriate instructional materials, provide inservice training 
and offer technical assistance to tutors on curriculum and instructional materials. 

Most local programs maintain in-house Ubraries with distribution systems and/or have 
relationships with their local Ubraries for tiiis purpose. These materials should contain aU the 
quaUtics Usted: appropriate for studem abUities; appropriate for student interests and needs and 
reflect diverse socioeconomic indications and culture of learners. There is an increasing 
availabiUty of commercial materials meeting these conditions. Programs are stUl required to be 
creative because of lade of resource to purchase these materials. 

Equipment that can be afforded should be selected and managed to provide Uie greatest 
access and applicability for the largest number of learners possible. Some method to determine 
this should be applied. Once purchased, the program should monitor and evaluate the same. 
PROGRAM OUTCOMES 
RetcBtioo 

The bottom Une for measuring program and learning retention is whether or not the 
learner achieves their goat Tliis assumes that the program determined the learner to be 
appropriate for service and vice versa. Post intake interview referraU to another, more 
appropriate program should be considered a positive program quaUty indicator. AduU education 
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should shift away from seat time retention criteria to goal achievement indicators - quality instead 
of quantity. 

Volunteer Uteracy programs typically ofifer one-to^ne tutorial instruction based on a tutor 
and learner commitment to two one-hour instrucUonal sessions per week. The average number of 
instructional hours per yew in our network is approximately forty. This number could become a 
benchmark for volunteer Uteracy programs. Qearly, if programs aren't achieving near this number 
of instructional hours per year, per learner something isn't going right and programs should 
question the output. In order to determine this average there must be a clear and consistent start 
point for all volunteer Uteracy programs. If we include learners who received one instructional 
hour then dropped-out, it wUl effect very different results than if we measure only those who 
receive tweWe or more hours as the Adult Education Act suggest. TTiere must be sector wide 
standards if this retention criteria will carry any meaning. 

As stated eariier. volunteer literacy programs must be concerned with retaining both 
learners and volunteers. Some standard and credit should be given for volunteer retention. 

Retention staadatds must be service sector specific. There is no way to compare a group 
instruction program structured on minimum of fifteen hours per week to a volunteer Uteracy 
program structured on two one-hour sessions per week based on hours of instruction received. 
Such criterion would serve only to eUminate resource from a field already in deficit All programs 
should monitor retention rates of population subgroups they purport to serve. Evidence of low 
retention rates among a specific subgroup warrant a re-cxamination of program capacity and 
output 

Educational Gains 

Educational gains which best lead to goal achievement are the most positive program 
outcomes. Hiere are. however, multiple issues with suindardized test scores which suggest that 
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this is not the optimal approach to verifying educational gains. Such issues include cultural bias, 
fear of test taking resulting in inaccurate assessment, test score increments that are so broad they 
don't reflect smaUer signiGcant gains. Competencies attained arc probably a better approach to 
determining learner and program outcomes however, these competencies must be achievable for 
the learner given their starting point In other words, a program that assists learner who cannot 
identify letters of the alphabet to be able to read a simple shopping list has achieved more than 
the program that achieves tiie same outcome with a learner who engages Uie program witii higher 
level reading sldlls. 

Programs should get credit for assisting learners to gain those prerequisite skflls to 
learning. Volunteer literacy programs are very invoNed in imparting tiiis type of skill or ability. 
These preconditions include self-esteem, the ability to learn and knowing how to learn. 
Traditional education programs focus on content or what to learn and assume that in learning 
content, learners wiU leara technique ~ wUl learn how to process information. Many adults 
considered iUiterate have never learned how to learn and cannot be judged on what they know 
until they are judged on if they know how to learn. Most standardized tests and compet<;ncy lists 
assume this basic precondition. 
Employment 

If employment related goals are consistent with the learners goals they represent positive 
program outcomes. For many this consistency wiU exist, especially for those currently employed 
or those witii skills which make them more nearly markeuble. Again, for programs serving adults 
who have not attained learning prerequisite skills (much less traditional educational gains), have 
multiple literacy related problems and live in areas of high unemployment, employment outcome-' 
ate not viable outcomes. This is the area where it is easiest to "substitute social and economic 
goals" for personal goals. Tlie skills being imparted by programs which are prerequisite to 
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emplpyabOity are needed tnd wUl determine the future employability of that iadividuaL Great 
caution should be applied to this criteria to insure that there is reasonable opportunity to achieve 
the outcome given economic conditions, and that the outcomes are learner consistent 
Goal Achiercinent 

Cioals. which arc agreed to by learner and practitioner as achievable and appropriate, arc 
the best indicator of program quality. It is the systtim's responsPoiUty to indicate the 
connectedness between these personal goaU and social and economic goals, not the learners'. 
When a system determines the standards for current and future social and economic participation 
we will be better able to integrate and assess how personal goals enable social and economic 
development Untfl that time it is difficult to hold programs accounuble for anything other than 

personal goal achievement 

TJusrtsponseu strictly the opinum of the auOtor. It does not necessarily reflect the thinking, 

phUosophy orproenm operation ofLVA-NYS, LVA orLLA. his intended to stimulate discussion 

and debate around the issues it addresses toward improvement in the volunteer service sector and the 

field. Thanks to Janice Cuddahee, Associate Executive Director, LVA-NYS and Chip Cariin, 

Director of Development A Information Services, LVA-NYS for their input and patience. 
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Resource Documents 



Pelavin Associates, Inc. prepared the following documents as part of this project and other 
activities performed under contract to the VS. Department of Education. All documents are 
available through the Division of Adult Education and Literacy aearinghouse, U.S. Department 
of Education. 400 Maryland Avenue, S.W., Washingtoc, DC 20202-7240, (202) 205-9996. 

Synthesis of State Quality Indicators for Adult Educadon Programs, by Larry Condelli, Judy 
Kolosld. and Lenore Webb. 

Model Indicators of Program Quality for Adult Education Programs, Office of Vocational and 
Adult Education, U.S. Department of Education. 

Quality Indicators, Measures and Performance Standards, by Larry Condelli. 

Primary and Secondary Indicators of Program Quality for Adult Education Programs, by Lany 
Condelli. 

Evaluation Framework for the State Adult Education Program, by Joel Sherman and Leny 
Condelli. 
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APPENDIX 
SAMPLE QUAUTY INDICATORS 
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SAMPLE QUALITY DTOICATORS 
FOR ADULT EDUCATION PROGRAMS 
USED BY STATES AND IN PROGRAM EVALUATIONS 



The foIkmiBg list of indicators ws provided «s a foide to the paper writers. The final 
model iadicatois for the project aro described in Modd Indicatan of Program Quality for AduU 
Education Programs, avaihible from the U. S. Department of Education, Division of Adult 
Education and Literacy Clearinghouse. 

PROGRAM CONTEXT 

1. Documented Need for Program Services 

A. Number and demographics of target populations in need 

B. Literacy levels in community 

C. High school dropouts in commimity 

D. Employment-related skill needs of community 

2. Organization and Structure of Delivery System 

A. Number of projects 

B. Variety of locations and settings of projects 
C Varied types of projects (e.g., ESL, GED) 
D. Flexible scheduling 

3. Characteristics of Participants 

A. Number and demographics of participants 

B. Number and demographics of participanu by sldll level 

C Number and demographics of participanu by program type 
D. Number and demographics of participants by program setting 
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PROGRAM PROCESS AND CONTENT 
1. PROGRAM PLANNING 



CommvBity and StaB Input in Pro{nm Development 
A. Existence and use of an «dvisory board 
6. Program holds public hearings 
C Use of staff input 

D. Other sources consulted (eg., employers, staff, CBOs. evaluations, program 

performance reviews) 

Cooidination Activities 

A. Existence of coordination arrangements (formal or informal agreements, agencies 

involved - number and type 

B. Typeof coordinated activities: referrals; 

share staff and/or facilities; 
joint planning and budgeting 

Written Operational Plan 

A. Existence of a plan 

B. Measurable goals and objectives specified 

C Specific program goals and objectives specified consistem with state plan 
D. Phin development process includes broad input and is open to change 

n. PROGRAM CONTENT 

Recruitment 

A. Recruitment methods used 

B. Special populations targeted 

C Program outreach and publicity activities 
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Procram InUke Procedures 

A. Entry policies 

B. Incoming assessment procedures 

C. Development of individual learning plans 

Ongoing Assessment Methods 

A. Procedures for monitoring student progress and learning gains 

B. Procedures for monitoring student progress toward goals 

Support Services 

A. Type of support services offered (e.g., counseling, transportation, child care) 

B. Adequacy of services for meeting student needs 
Evaluation 

A. Student, community and staff evaluation of program activities 

m. STAFF QUALinCATlONS AND RESPONSmiUnES 

Characteristics of Staff 

A. Demographics 

B. Educational background, credentials, experience 
C Number of staff 

D. Staff retention 

SufrRespoBsibUitics 

A. Duties of stafif appropriate for position 

B. Staff commitment (e.g., fuU or part-time, additional duties) 
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3. Staff Development 

A. Existence of staff development component: 

1. When provided (pre- or in-service) 

2. Content, topics covered 

3. Duration 

4. Sequential training 

5. Staff compensation for attendance 

B. Evaluation, of staff development activities: 

1. Systematic needs assessment for content 

2. Evaluation of activities by staff 

3. Staff participation in development 

4. Use of Volunteer Staff 

A. Duties of volunteers 

B. Volunteer training 

5. Evaluation of Staff Performance 

A. Methods for evaluating staff 

IV. CURRICULUM AND MATERIALS 

1. Type of Cnrriculum and Instruction Used 

A. Instructional methods meet student needs (sequential, individual, competency-based) 

B. Topical emphasis relevant to adult learners 

C Instructional technique (e.g, peer teaching, small group) 

D. Amount of instruction offered 

E. Organized sequence of courses used 

F. Indwidualizcd instruction based on assessment 
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2. Materials and Equipment Used 

A. Adequate materials used: 

1. Appropriate for student abilities 

2. Appropriate for student interests and needs 

3. Reflect diverse socioeconomic and culture of learners 

B. Adequate equipment used: 

1. y^propriate to meet program and learner needs 

2. SufGcient amount to meet program and learner needs 

3. Selection and Evaluation of Materials Equipment 

A. Method Used to select and evaluate equipment and materials (e.g., instructor 
student input) 

PROGRAM OUTCOMES 

1. Retention and FoIIow*up Methods 

A. Hours of instruction received 

B. Participation rates for population subgroups 
C Methods for contacting program leavers 
D. Exit interviews conducted 

2. Educational Gains 
A Grade level advancement 

B. Competencies attained 

C. GED or high school graduation attained 

3. Employment 

A. Attained new employment 

B. Improved current employment 
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C Improved income 

D. Attained employment-related skills 

E. Receiving public assistance 
Personal and Social Goal Achievement 

A Achieved personal goals for participation 
B. Improved self-esteem and self-confidence 
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